This is great but we really need an audio-to-audio model like they demoed in the open source world. Does anyone know of anything like that?
Edit: someone found one: https://news.ycombinator.com/item?id=40346992
Edit: someone found one: https://news.ycombinator.com/item?id=40346992
We should probably update the example in the README to better represent that, thank you!
But..I looked at the code but didn't see any audio-to-audio service or model. Can you link to an example of that?
I don't mean speech to text to LLM to text to speech. I mean speech-to-speech directly, as in the ML model takes audio as input and outputs audio. As they have now in OpenAI.
I am very familiar with the typical multi-model workflow and have implemented it several times.
For context relating to real-time voice AI: once you're down below ~800ms things are fast enough to feel naturally responsive for most people and use cases.
The GPT-4o announcement page says they average ~320ms time to first token from an audio prompt. Which is definitely next level and is really, really exciting. You can't get to 800ms with any pipeline that includes GPT-4 Turbo today, so this is a big deal.
It's possible to do ~500ms time to first token by pipelining today's fastest transcription, inference, and tts models. (For example, Deepgram transcription, Groq Llama-3, Deepgram Aura voices.)