Well technically it is possible to avoid the signaling server, but you'll need some other channel to exchange the peers connection data. This is pretty inconvenient though. But a signal server is a rather light thing and easy to install on a location you trust: https://github.com/holtwick/briefing/tree/master/signal