You can also write it in C and use lthread. It will be faster than what Go has to offer.
But ideally you don't want to allocate memory at all. You want to have memory preallocated and preferably accessing a page that's still in cache. You can control memory pages in C because it doesn't manage that for you.
You meant “could theoretically be” – you're gambling that you'll have development resources and expertise to make optimizations which aren't possible in Go, that those optimizations will actually matter to the critical path, and that you're more likely to reach the point of being able to make those optimizations than the Go team will be able to optimize further.
In the absence of something showing a strong, unavoidable disadvantage for Go, that's a lot to take on faith.