We turned EmbeddingGemma 2 into a speech-to-text model
eg2.haku.sh
eg2.haku.sh
Just to clarify - This started as just an experiment to see if we can use the fact that text and audio share a vector space in Embedding Gemma 2 to convert audio to text. Had to hack around a few things, but, it worked. Speed was just a side benefit. Was not aiming for it.
The accuracy is good, but not in the range of cloud models, but decent enough for everyday use, and this is just a couple of days work. The WER can improve a lot with more training on varied data. Will try that next.
I will opensource the whole thing soon. Might need a few days though. Also, releasing the TTS version for this soon.
Happy to answer any questions you have.
Basic ML is probably even cheaper. But LLMs might have higher accuracy. “Uhh one I think” = 1
Say classify recorded interviews into viable/not viable candidate?
But the one-shot and few-shot classification performance of decision models really needs in-context learning. The most common way of getting that is by grafting a small LLM onto it, which is certainly possible.
At that point though, why not just use a small LLM with audio capabilities. Gemma E2B is a good model that is already setup for this, and training a decision model LoRA adapter is very easy.
Edit: Apparently people have got decision models working on EmbeddingGemma, eg: https://huggingface.co/mrm8488/embeddinggemma-2-ft-typed-dec.... I'm not seeing it ranked on https://huggingface.co/spaces/multimodalart/jev-decision-ind... so I have questions about how well it does in-context learning.
People aren't paying enough attention to how great EmbeddingGemma 2 and Gemma 4 are.
Thank you!
uv tool install mlx-audio
mlx_audio.stt.generate \
--model ibm-granite/granite-speech-5.0-470m-turboctc \
--audio.wav \
--output-path transcript
"One cool trick" I like for running those kinds of models very quickly on macOS is converting them to CoreML so they run in the Neural Engine.
I had amazing success with Spleeter, Demucs, MVSEP-MDX23-music-separation, DeepFilterNet, Whisper, Hunyuan3D for 3D mesh generation. Plus a lot of others for both work and leisure.
I also convert the code to Swift without dependencies, but that's a personal preference.
The runtime on some of them went sometimes from 60s to 10s. And since it's just Swift, I can just give them to non-technical co-workers and they don't have to setup a whole Python environment.
If you don't wanna do this by hand, this is something that existing LLM models are good at. Even Luna can do it without much fuss.
Although I'm sure someone is bolder than me is running a fleet of Mac Studios somewhere!
Pre-pandemic it seemed like even the very best in class auto-transcription features were just unusably bad. I guess that was like 7 years years ago though.
I'll test theirs too if I get around to it
Here's a sample CLI wrapping the apple model and providing a simple benchmark: https://github.com/llimllib/transcribe
update: doesn't seem like they provide their model and I don't feel like trying to dig it out of their app bundle, so I'm not going to test this one
I use their on-device translation in my apps. Works pretty well. ChatGPT's context-aware translation is better, but it's not on-device.
They supposedly improved it on later Apple Intelligence enabled devices, but I don’t have one.
On my bench, parakeet-tdt-0.6b-v2 is 2x slower
I wonder if you're using different Apple models when you go through that app? I don't know
"EmbeddingGemma 2 is made by Google DeepMind and released under Apache 2.0. This is an independent project, not affiliated with or endorsed by Google."
Does this work for Hindi or other non English languages though?
I wanted to recently transcribe Hindi + English (mixed) Audio and I used this model[0] and created a custom transcribe command which can give text over input of mp4 files
This took me roughly around an hour or more or around this time frame for a 90 minute transcription source. I would consider myself to be slightly bored during this phase.
I would really appreciate it if something like the project could be made native for a lot more languages but I think that the approach of embedding models and using them in such a way is pretty crazy. Kudos for it and I will try it out!
I can think of many such use cases where Speech to text of even large audio files becomes near instant. As I think someone on hackernews once quoted whom I am very paraphrasing
An order of one or two magnitude of an already existing thing stops becoming a quantity upgrade but rather becomes a quality upgrade in it of itself as it provides access to workflows which previously wouldn't have been feasible.
[0]: https://huggingface.co/parismitaglobalsolutions/indicconform...
You stated WER of 5% - against which benchmark?
Whistle [1] for example is a 16MB STT model but WER varies between 4-26% depending on the benchmark.
I'm guessing that would make it a good contender as a sort of all purpose tool for STT / TTS / Translation?
That could be super cool, right now it is really annoying to have to juggle between different models, all of which have different capabilities depending on the language.
Library/Application\ Support/ai.haku.eg2demoOr is there another approach?
My point was that the "natural speed" of video (i.e. the frame rate - clear slow motion requires more frames, while fast blurred motion will generally have fewer) does actually impact how much data must be processed. It's true of audio too (sample rate) and downsampling is a common optimization technique.
so you have a file containing bytes representing the sound wave, and you have existing recordings (read: sound waves represented as sequences of bytes) of people speaking in different languages and contests, and you can feed all those existing recordings into a model along with corresponding transcriptions, and you can get a model out which can identify patterns in recordings of people speaking to then go backwards and see a pattern of sound waves and correctly associate it with a specific token, no need to play anything back. pauses, stutters, etc. are themselves patterns of sound waves that can be identified and worked around to identify just the meaningful parts of speech.
in principle, you could probably learn to do this too, by sitting down for hundreds of hours meticulously starting at waveforms, and this is, very roughly, what signal analysts do: look at waveforms and try to identify patterns. ai just leverages this principle to turn mundane speech into text.
But the overall transcription quality is very low.
My reaction to seeing the project was that being an embedding model it'd be pretty good at conveying the meaning but not great as using the same terms, is that what you're seeing?
Otherwise could run two passes to get down to .25% or three passes to .0125%. Or four passes to .000625%!
That’d be .02 seconds of error per hour!