Show HN: Live speech-to-text on an ESP32-P4 microcontroller, no cloud | Hacker News Reader

Show HN: Live speech-to-text on an ESP32-P4 microcontroller, no cloud

github.com

Kroko ASR team here, great job and thanks for the shoutout! We have some more models coming that might be helpful, I will contact you.

Nice! Thanks for posting. How critical is the 1.28s chunking? Is that to ensure getting a full word? I'd think getting final-word latency down would be a goal. Does the 0.9s processing delay scale down with the sample size?

Median of less than 2 seconds for final word after its spoken on esp32 is insane, amazing work!

That's very impressive! I haven't heard of Kroko, Banafo, or Zipformer2 before.

Thanks! They're not well known outside speech circles. Zipformer is an encoder architecture from the k2 / icefall team (the next-generation Kaldi project), made for fast streaming speech recognition: https://arxiv.org/abs/2310.11230

Banafo is a small company that trained Kroko, a streaming Zipformer2 transducer, and released community models on Hugging Face: https://huggingface.co/Banafo/Kroko-ASR

I picked a transducer because it suits a microcontroller. The encoder is 98% of the weights, and it runs once per 1.28 s chunk of audio, so each weight is read from PSRAM once and reused for every frame of the chunk. The decoder is tiny (under 2% of the model) and only remembers the last two tokens. On these chips the limit is reading the weights from memory, not the math. A Whisper-style model is the opposite: most of its weights are in a decoder that runs once per token, rereading them every time.

This looks interesting!

What's the power draw while transcribing?

Hi HN, I'm the author. This runs an open-vocabulary English speech recognizer live on an ESP32-P4: you speak into the board's mic and the words appear on its screen as you talk. No network, no NPU: just the chip's two RISC-V cores and their SIMD extension (PIE).

The model is Kroko (by Banafo), a 65.7M-parameter streaming Zipformer2 transducer. As released it is 71 MB in int8, which does not fit the board's 32 MB of PSRAM, so I quantized the weights to 2 and 4 bits (26.65 MB, groups of 128 with a Hadamard rotation, the bit width chosen per matrix) and wrote my own engine: integer arithmetic throughout, with the hot loops hand-written in PIE assembly.

Numbers, measured on all of LibriSpeech:

test-clean 4.34% WER, test-other 10.26% (the original int8 model on a PC: 3.94% and 9.17%).

1.28 s of audio processed in 0.90 s (0.70x real time), both cores at 400 MHz.

A word is final about 2 s after it is spoken.

One thing I care about: the chip computes exactly the same bits as the engine's PC build, so every accuracy number above is the chip's, not a simulation. Getting there meant replacing libm (newlib and glibc disagree in the last bit of log/exp/cos) with deterministic math on both sides.

There are also hotwords (phrases to listen for, from a text file on the SD card) and six voice-control demos driven by simple word rules, no LLM.

Limits: English only; it needs the chip revision v3 (P4X) with 32 MB PSRAM; accuracy is measured on clean read speech, noisy rooms not yet. You can flash it from the browser; the weights are on Hugging Face.

Video: https://www.youtube.com/watch?v=oe-2MowdsSs

Happy to answer questions about the quantization, the kernels or the streaming.

Reply on news.ycombinator.com