Also consider using Cerebras' inference APIs. They released a voice demo a while back and the latency of their model inference is insane.
Also Groq is very fast, but the latency wasn't always consistent and I saw some very strange responses on a few calls that I had to attribute to quantization.