Show HN: The First Open Source Diffusion ASR Audio Model 15x Faster Than Whisper
arxiv.org
Instead of generating text one token at a time like Whisper, in diffusion-gemma-asr we transcribe by denoising an entire transcript in parallel.
To enable the model to learn, we don’t touch DiffusionGemma in training, rather we use a frozen whisper-small’s encoder for acoustic features, and train only a 42M parameter adapter, about 0.16% of the entire model.
We keep both the Whisper-small encoder and DiffusionGemma’s original weights frozen. And train a convolutional projector that compresses the Whisper’s 1,500 audio frames into 188 audio tokens, along with small LoRA adapters on DiffusionGemma’s attention layers so the model can learn to route those new tokens.
DiffusionGemma is different from most diffusion language models as it does not begin with a sequence of mask tokens. It begins with a fixed canvas filled with random vocabulary tokens. At each step it keeps the positions it is confident about and replaces the uncertain positions with fresh random tokens until the canvas settles into a transcript.
We pass the projected audio embeddings through DiffusionGemma's frozen language head and optimize them with a CTC loss. That gives the projector a learning signal without relying on attention. Once the embeddings became meaningful, attention naturally started using them, and the model actually started transcribing audio.
The CTC loss broke that loop where model produced repeated english stop words by supervising the projected audio directly through DiffusionGemma’s frozen language head, without requiring attention to use it first. And as those audio embeddings began predicting transcript tokens, attention naturally started using them and the model began transcribing. CTC is only a training scaffold and it is removed entirely during inference.
Upon completion of training on Librispeech, english WER dropped significantly on clean test set. We then warm started from that checkpoint and finetuned on FLEURS across English, German, French, Spanish, Hindi, and Mandarin, later mixing in VoxPopuli to improve ASR on accented speech.
On Librispeech test set for english, our model sees 6.6% WER, compared to 8.3% for Whisfusion and roughly 7% reported by TransFusion, while using a smaller encoder, making us SoTA for Diffusion-ASR .
We're still behind standard Whisper, but that's not surprising. As Whisper was trained on millions of hours of speech. Our Gemma variant has seen roughly 219 hours. The architecture seems to scale; we simply haven't trained it on enough audio yet.
The biggest takeaway from this project was that a frozen diffusion LLM can learn a completely new modality by training only a small adapter as long as you give that adapter a way to learn before the rest of the model decides to ignore it.
The adapter weights, code, inference script, and model card are all open source. We’re Interfaze, a research lab building deterministic AI models. You can try it here: https://huggingface.co/spaces/interfaze-ai/diffusion-gemma-a...