You need a server for holepunching/STUN via IPv4, but not for IPv6 since each client has a unique IP instead of being behind the router's address.
I didn't see that you mentioned UPnP in the OP though, sorry. I'd assume downloading metadata from a signalling server if you don't need it for traversal is completely optional - most P2P networks have an initial list of peers to connect to to bootstrap new clients.