While not actually speeding things up, I wonder if it's possible to run a very small autocomplete model to guess what the next sentence ahead of time from the LLM on the client side. To at least keep the perceived latency low.

Thus if there is a delay, you still get the stream of text coming at a steady rate, but you may have to backtrack and correct for mispredictions.