Show HN: RealtimeVAD – trading false positives for 110ms earlier end-of-speech | Hacker News Reader

Show HN: RealtimeVAD – trading false positives for 110ms earlier end-of-speech

github.com

31 pointskoljab8 comments

Hey guys. I recently watched the latency reports for my local voice assistant, which show a waveform plus silero probability for voice activity. I noticed silero lags visibly on the speech off signal.

There had to be a method to get this signal closer to what I felt is optimal. My idea was to use a 10 ms energy detector with a threshold that adapts to environment or room noise. Which worked amazingly well. It created more false positives but in my speculative generation pipeline that would not matter. If it's wrong and the user continues speaking I can abort the pipeline fast.

I benchmarked my solution over ~3200 turns from my assistant logs and it came out ~110 ms faster initially. Still 100 ms now after some a fix which was needed for quiet speech.

I made it public if you want to have a look. If you measure it on your data I really appreciate any feedback. There's a benchmark script in the repo.

Thx for reading, have a good day folks.

Open articleView on HN

You can also avoid the problem all together. I made a "speculative regeneration" at least thats what i call it pipeline i been using for many months now and it works great. basically as long as you have access to the inference server you are good to go. idea is as such, every n milliseconds of silence (usually 300ms) for me, you send the asr transcript for the llm to process immediately. you let the model start answering , but if the user starts talking again before the end of turn token is reached you cancel the generation and again wait for the 300ms. rinse repeat. there is a safety window you have to set up as well that way the final llm answer gets finalized regardless of its inference speeds. works like a charm. basically the user gets 300ms latency at the highest for that "silence" window. so the only other thing you have to worry about as faras latency is llm inferance speeds and asr and tts latencies. in my pipeline the moment the user stops talking, the model ,is already responding within about 900ms - 1.2 sec.. its my gemma 26b moe model , 4 bit quant thinking on... with mtp getting about 300 tokens per second... asr and tts are both under 350ms for total... anyways, speeds could be improved if turning off thinking but i like my voice agent to be smart as he has tool calling and common sence capability is reduced when thinking is off.... fun times!

This is the 3rd or so time I've posted this -

Run a tiny shitty llm, one that is super fast. Or better yet try out a decision model now days.

Feed it the previous conversation turn and all the STT (ASR output) you have so far and just ask it for a probability that the user is done speaking.

Use that probability to shorten your VAD time.

It isn't as good as a true speech to speech model but this will cut a few hundred ms compared to pure VAD. Oh as a bonus it also lets you wait longer when the user isn't done talking yet.

Thanks I had not heard of that idea yet. Especially like it for the waiting longer part when I am not done talking.

It's quite cool. Look into Semantic End-of-Turn Detection, also called Semantic Turn Detection or Semantic VAD.

Oh it has a name, cool!

I first did this around 2023 because I needed to make things fast and I was out of other options!

I had tried language models but seemed slower, which model responded faster in your app?

Whatever is cheap and has less than 20b parameters.

It really doesn't matter, asking "is this a complete reply to the previous statement" is hardly an earth shatteringly difficult question for any LLM made in the last 3 years.

which model works fastest for you?

Reply on news.ycombinator.com