There are 2 issues here. The first is that speech to text is a lot more useful if it can run realtime which puts upper limits on model size. The bigger reason however, is probably that there's a lot less data for text to speech.