Yeah, you'd need a proxy server (websockets) to establish WebRTC connections. Even after a WebRTC connection is established, you'd need a STUN or a TURN (coturn supports both) server to get around NATs. Pls somebody correct me if I'm wrong!
The biggest issue is that WebRTC connections do not allow you to send raw data over TCP or UDP[1].
That being said, it shouldn't be too difficult to just create a WebSocket server that translates between the native network protocol and HTTP WebSockets[2].
[1] https://developer.mozilla.org/en-US/docs/Web/API/RTCDataChannel
[2] https://wiki.openttd.org/Network_Protocol