I ask because I know Opus is used quite commonly in real-time applications (e.g. voip), and I remember when researching in the past that it is actually capable of lower inherent latency because of support for smaller frame sizes than some of its competition.
I haven't looked into how expensive it is to encode in terms of CPU time, so I assume maybe you're taking about a bottleneck in terms of the number of simultaneous streams you can support on a single CPU?