Go isn't the problem. The issues I have seen is that you are I/O bound by the kernel.
One company did Pion WebRTC + https://www.dpdk.org/ and is now doing 4x throughput. They tried switching to Rust (and still doing userspace networking) and it didn't make a big difference.