Host Your Own Web Conferences

I have been looking around at the open-source solutions to self-host your own web conferences as an alternative to commercial solutions. I first looked at Big Blue Button and later discovered the project Jitsi. Both offer screen sharing, audio/video/webcam sharing, external videos, public/private chat, and emoticons. Jitsi supports multiple screen shares while BBB supports document upload, whiteboards, polling and breakout rooms. BBB recommends Ubuntu 16.04 and a beefy server, while Jitsi runs on Ubuntu 18.04, with fewer resources and is much easier to set up. For most conference use cases, I recommend Jitsi. But if you need classroom-like features such as a whiteboard and slides, breakout rooms, or require a front-end with user management, BBB is the way to go.

Update: April 6th, 2020

I had an opportunity to stress-test Jitsi this weekend with family. We had around 10 users and after some time the quality starts to degrade. When I use tiling view especially, it would start blacking out video feeds unless the user was talking, and sometimes it would just stay blacked out entirely. It can also be tricky on mobile users to convince them to either download an app (which I haven’t tested) or to tap settings and reload with "Desktop Site" checked.

Meanwhile, I tried to get the group to migrate to Big Blue Button to test multiple users sharing webcam on there. Big Blue Button worked well with 4 users, but then my brother couldn’t access it from his iPhone, despite Safari being listed as supported. Not sure what was up with that, but it seems iPhone support is spotty.

Why Self-Host?

There are a lot of commercial conferencing solutions that have made their products available as the global corporate workforce migrates to remote work during the COVID-19 pandemic.

Zoom is one such solution; by offering their product for free, they have succeeded in becoming an ubiquitous household name for remote conferencing. They even have a browser-only version for those of us who don’t trust the security of their client. But the video quality and the features of the conferencing can be limited, and the conferences only last 40 minutes at the free tier. There are also privacy concerns and poor default configurations that has enabled "zoom bombing."

Cisco Webex also now offers a free tier for conferencing. Like Zoom, they also have a conferencing solution that is browser-only, even though they also provide a client to download and install. I did check out a browser-based meeting and it looked alright, but it’s also Cisco Webex, so it probably has the same issues as Zoom, but worse.

I’m sure there are other options out there; in fact, while I was writing this post, Skype announced a free offering. If you need a conference in a pinch, these can work. Personally I would avoid using these for things that were sensitive, such as a country’s pandemic response cabinet meeting. The purpose of this post not to dissuade people from using these, but to explore self-hosted solutions that gives the user more control over privacy and security.

Big Blue Button

I first discovered Big Blue Button when I learned the DOD is using it for its Defense Collaboration Services web suite. It looks like the DOD is stuck on an older version which requires java and flash, but BBB itself now runs a full-featured HTML5 browser client with no external dependencies.

Out of the box, with just server and a modern web-browser, BBB supports the following:

  • Chat, both public and private
  • Webcams (all users simultaneously)
  • Audio sharing
  • Emojis
  • Breakout Rooms (up to 8)
  • Screen Sharing (one user at a time)
  • Multi-user whiteboard with document import support and annotations
  • Polling
  • Raising hand
  • Custom status (beyond raising hand)
  • Shared Notes
  • Meeting Recording

Security

When you don’t look under the hood, BBB is a fantastic solution for many use cases. That said, I would not install this on a network I wanted to protect, nor trust it to share things I wouldn’t send over SMS.

To give some examples, they only support Ubuntu 16.04 LTS and probably won’t even start working on 18.04 support until 20.04 is out. Their all-in-one installer script uses NodeJS 8, which is at end of life and does not receive security updates. If you run the script as-is, Node gives you scary warnings.

I was able to edit the installer to use Node 10 instead of Node 8. It probably would work with Node 12, I didn’t test. My best guess was that there were fewer changes between 8 and 10 vs. 8 and 12, but I don’t really use Node so I’m not informed enough to know.

I installed this on a standalone droplet in Digital Ocean, set up unattended upgrades, threw up firewall rules, and called it a day. I also saw a suggestion on Twitter to snapshot and wipe in between uses. If you’re the sole user of the server this can work, but it might wipe other user’s configurations if you are sharing this with others.

Installation

Let’s assume we’re starting with a fresh Ubuntu 16.04 server; here’s a guide from Digital Ocean that can help you get there.

Per their documentation, you want at least 8GB RAM and 4 CPU cores. I tested on 16GB RAM and 6 cores on Digital Ocean with minimal issues hosting dozens of rooms, and a few with around 20 users.

The documentation also notes that SSL is required if you want WebRTC to work with modern browsers. I assume this is correct, and it looked accurate when I googled it.[1]

Their guide gives you manual step-by-step instructions, but also has an installer script. They recommend, as root, using the installer like this:

wget -qO- https://ubuntu.bigbluebutton.org/bbb-install.sh | bash -s -- -v xenial-220 -a

I’m not a fan off piping things from the internet to a root shell. I grabbed the installer and gave it a quick look over in vim. I edited the node line and changed this:

if [ ! -f /etc/apt/sources.list.d/nodesource.list ]; then
		curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
fi
if ! apt-cache madison nodejs | grep -q node_8; then
		err "Did not detect nodejs 8.x candidate for installation"
fi

To this:

if [ ! -f /etc/apt/sources.list.d/nodesource.list ]; then
		curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash -
fi
if ! apt-cache madison nodejs | grep -q node_10; then
		err "Did not detect nodejs 10.x candidate for installation"
fi

I then launched the installer using the example SSL + Greenlight options:

bash bbb-install.sh -v xenial-220 \
-s bbb.example.com -e info@example.com -g

Replacebbb.example.comwith the domain pointing to your instance, andinfo@example.comto whatever email you want Let’s Encrypt to publish. The script will download and set up everything, and then configure SSL via Let’s Encrypt and install Greenlight on top of it.

It takes a while for the installer to do its thing but it works fairly consistently on a clean install of Ubuntu 16.04.

Once it is installed, you can manage it with Greenlight: a Ruby on Rails web front-end they provide to manager the server. It runs in its own docker container that gets dropped wherever you run the installer.

You can add your own admin user in the docker container via:

docker exec greenlight-v2 bundle exec rake \
user:create["username","e@mai.l","password","admin"]

If you reboot your VM and you don’t see greenlight come online, you’ll need to start the docker container. By default it doesn’t do this:

cd greenlight
docker-compose up -d

Finally, set up your firewall rules. In UFW, this looks something like:

root@bbb:/# ufw status
Status: active
To                         Action      From
--                         ------      ----
22                         ALLOW       Anywhere
Nginx Full                 ALLOW       Anywhere
16384:32768/udp            ALLOW       Anywhere

NAT Configuration

If you are hosting in somewhere like Azure, where you literally can’t assign a public IP directly to a VM[2], you will have to walk through the additional instructions for firewalls and NAT.

Additionally, when installing on a host which is NAT’d, I had to manually comment out the check that verifies the IP in the DNS configurations for the domain (the public IP) and the LAN IP are the same. This line comes from thecheck_host()function:

#if [ "$DIG_IP" != "$IP" ]; then err "DNS lookup for $1 resolved to $DIG_IP but didn't match local $IP."; fi

In this configuration, I did not have to have to use an STUN server. Instead, I edited/etc/kurento/modules/kurento/WebRtcEndpoint.conf.iniand explicitly set my WAN_IP like this:

externalAddress=WAN_IP

Otherwise, I followed the guide to update freeswitch according to their documentation. I did not install a dummy NIC.

A final note with NAT and specifically in Azure, you will want to add an additional rule set, and all of the firewall rules should be reflected in your security group:

7443/tcp                   ALLOW       LAN_IP
7443/tcp                   ALLOW       WAN_IP

This second set of rules are necessary because it tries to talk to the freeswitch component on its public IP, which gets blocked if you don’t explicitly allow it in your firewall and security group rules. I suppose this is probably a configuration issue with freeswitch that I could have changed, but I didn’t have a lot of time to mess with the Azure instance.

Testing It Out

Log in with the account you created (or make a separate non-admin user) and join your default room:

web conf 73dd4

Once you join the room, it asks you how you would like to set up audio:

web conf 77196

When testing, join with a microphone; it will connect to the WebRTC echo test server and do an echo test with your mic. If WebRTC is working correctly, you should hear audio.

If it connects but you don’t have audio, that’s probably something specific to your browser sharing the audio and not with the server itself.

Look around and test out screen sharing, webcam sharing, chat, notes, user status and so on.

web conf fe199

You can control polling, upload presentations, add videos, and control camera and screen sharing on the bottom:

web conf 79830

Whiteboard control is on the right side:

web conf 793de

You can manage individual user controls, create breakout rooms and configure closed captioning here:

web conf 66a0c

After some uptime, I noticed that sometimes the WebRTC components get stuck.

Sometimes the error isMedia server requests are timing out (error 2003). I found a fix for it in this issue:

systemctl restart kurento-media-server bbb-webrtc-sfu

Jitsi

Jitsi is an open-source project that has been maintained by Atlassian, until it was recently sold by Atlassian to 8x8. On their website, Jitsi describes itself as:

A crazy-good, completely free video conferencing solution that anyone can use.

Some features they list that set it apart:

  • Unlike other videoconferencing technologies, Jitsi Videobridge, the heart of Jitsi, passes everyone’s video and audio to all participants, rather than mixing them first.
  • The result is lower latency, better quality and, if you are running your own service, a much more scalable and inexpensive solution.
  • Jitsi is compatible with WebRTC, the open standard for Web communication.
  • Jitsi supports advanced video routing concepts such as simulcast, bandwidth estimations, scalable video coding and many others.
  • Ubuntu and Debian packages for easy installation

In comparison to BBB, there is no slide import, whiteboard, or breakout rooms. The interface feels very much like Zoom.

However, they do allow all members to screen share, which is different than the presenter vs. attendee model that BBB has.

Installation

Let’s assume we’re starting with a fresh Ubuntu 18.04 instance; if you need help getting there, I recommend this Digital Ocean tutorial.

The installation instructions listed on jitsi.org are extremely simple:

#add repo key
wget -qO - https://download.jitsi.org/jitsi-key.gpg.key | sudo apt-key add -

# add repository
sudo sh -c "echo 'deb https://download.jitsi.org stable/' > /etc/apt/sources.list.d/jitsi-stable.list"

# install
install sudo apt-get -y install jitsi-meet

When it finishes installing, it will spit out a script to run to configure Let’s Encrypt for your SSL:

You can now switch to a Let’s Encrypt certificate. To do so, execute:
/usr/share/jitsi-meet/scripts/install-letsencrypt-cert.sh

Finally, some firewall rules that seemed to work for me:

root@jitsi:~# ufw status
Status: active

To                         Action      From
--                         ------      ----
22/tcp                     ALLOW       Anywhere
80/tcp                     ALLOW       Anywhere
443/tcp                    ALLOW       Anywhere
1000/udp                   ALLOW       Anywhere

Once you’re up and running, here’s what it looks like:

web conf 04198

Note some additional features:

  • ability to adjust all of the stream quality
  • can control other people’s audio volume
  • everyone can share their own screens
  • everyone can share webcams
  • ability to switch between users in a tiling view or a single presenter view
  • no user management required. Just host a meeting and you’re the moderator for that session.

I did notice that holding a meeting of 4 participants at full quality was pretty awful while tethering from my phone at around 3Mbps. But I was able to turn down the quality to voice-only and had no issue.

Etherpad Configuration

While I didn’t really dig into BBB extra configurations, Jitsi listed etherpad integration on their main website and it looked really easy to configure so I decided to check it out. Based on some of the comments on this BBB issue, it looks like BBB might already be using etherpad under the hood.

The documentation for etherpad integration is fairly straight forward: a few commands to install etherpad, and then add a line to configure it to be used.

The recommended installation uses the gross nodejscurl | bashinstallation, but here goes anyway (in fact, the BBB installer does this on its own automatically):

#grab and install node
wget https://deb.nodesource.com/setup_10.x
bash setup_10.x
sudo apt install -y nodejs
#add a user for etherpad
adduser etherpad

#run etherpad
su etherpad
export NODE_ENV=production #this sets it in production mode
git clone --branch master https://github.com/ether/etherpad-lite.git && cd etherpad-lite
bin/run.sh 2>&1 1>etherpad.log &

That’s a pretty dirty way to run it, and it won’t survive reboots. You probably want to containerize it or something if you plan to keep it around for longer.

The final steps are documented here to add an entry into your nginx config and then add it into the jitsi json config:

/etc/nginx/sites-enabled/example.domain.conf

# Etherpad-lite,
# I put this in the server listening on 4444,
# which appears to also listen on 443
location ^~ /etherpad/ {
				proxy_pass http://localhost:9001/;
				proxy_set_header X-Forwarded-For $remote_addr;
				proxy_buffering off;
				proxy_set_header       Host $host;
}

/etc/jitsi/meet/example.domain-config.js

var config = {
    etherpad_base: 'https://example.domain/etherpad/p/',

Once installed, here’s how it looks:

web conf 73696

Overall, the Etherpad integration seems comparable to the shared notes feature in BBB, but it takes up the entire screen rather than sitting off to the side.

Final Thoughts

Jitsi wins on polished installation, user experience, and server specifications, while Big Blue Button has a better classroom support and seems to have somewhat lower bandwidth requirements. I rounded up on the spec requirements and have had success using Jitsi on a $20/mon droplet, while I have BBB in an $80/mon droplet.

From a security perspective, Jitsi feels better due to the polish and support for Ubuntu 18.04, but I don’t really know if it’s actually any more secure. I would keep both of them isolated and carefully monitor them. If you can, snapshot and restore them on a daily basis.

Hopefully you found something useful in this post. If you have any questions or suggestions, feel free to ping me on Twitter.

Glossary:

WebRTC
Web Real-Time Communication; according to Wikipedia, it is a free, open-source project that provides web browsers and mobile applications with real-time communication (RTC) via simple application programming interfaces (APIs).
SMS
Short Message Service; or simple text messaging via mobile devices. SMS is un-encrypted and vulnerable to spoofing. It is not safe for sharing your secrets, and probably neither is a free, open-source conferencing solution.
NAT
Network Address Translation; in this context, the public IP is being translated to some private internal IP that the VM sees.
STUN
Session Traversal Utilities for NAT; according to Wikipedia, this is a standardized set of methods, including a network protocol, for traversal of network address translator gateways in applications of real-time voice, video, messaging, and other interactive communications. As far as I can tell, when behind a NAT, it can be used by BBB to determine what the public IP is.

Footnotes:


  1. while Googling, I found that webrtchacks.com article that talks about how to bypass the new SSL requirements by proxying through localhost, which is a pretty bold strategy.
  2. the fact that this is the case still makes me mad as I write it