I also tried to hack together WebRTC headlessly a while ago actually! I looked at GStreamer and Kurento, but ended up using Janus instead. Anyway, turns out you can use Xvfb, essentially a "fake screen", to headlessly run GUI programs: https://en.wikipedia.org/wiki/Xvfb , which is what we're doing with Firefox.