You still need a proxy server, unless the OpenTTD game browser supports WebRTC connections.
The biggest issue is that WebRTC connections do not allow you to send raw data over TCP or UDP[1].
That being said, it shouldn't be too difficult to just create a WebSocket server that translates between the native network protocol and HTTP WebSockets[2].
[1] https://developer.mozilla.org/en-US/docs/Web/API/RTCDataChannel
[2] https://wiki.openttd.org/Network_Protocol