Do you still need a server for signaling or is it actually P2P at that point?
There needs to be some mechanism to tell you what the other P2P endpoints are and what address/port pairs they are listening to. It doesn't have to be a server necessarily, could be just a copy-paste string.