One likely scenario is that you've issued 128 RPCs to some other services and are waiting to hear back. Even if each RPC is, say, on a separate TCP connection, your network stack can handle plenty more.
> Also, what overhead do you see as the main limit on spawning a lot of threads? Is that the CPU time of context switching? If so, in this scenario CPU is not the bottleneck, and switching between processes will be nothing, especially with a 32-64 core CPU.
I don't remember what specific feature of OS threads contributes the most to overhead, and maybe someone else can answer this better. But context-switching burdens both the CPU and RAM (due to saved stacks).
> This is a genuine question
I always assume this anyway. Maybe not on Reddit ;)