626 karma · joined May 22, 2017
https://insidevoice.ai/
> I would expect an "AI Note Taker" to faithfully transcribe the entire conversation. With the same quality I see in a lot of automated video subtitles.. ie they use the wrong word a lot but it's easy to tell what they mean by context.
That's a reasonable expectation, but would not be a safe one. All transcription tools are not made the same. First it depends on what kind of STT/ASR (speech-to-text / automatic speech recognition) model they are using. A lot of tools like to use some flavor of OpenAI's Whisper model. It works well generally but I would never use it in a critical use case like healthcare. Because it can hallucinate. That's specific to its architecture and how it was trained.
There's a fairly large variety of architectures that can be used for STT/ASR. Some of them are designed for "offline" / "batch" / pre-recorded audio. Some are designed for fast real-time streaming transcription.
There are more factors too like training data. And not just demographics of the speakers in the training data but audio environments too. Was the model trained on echo-y doctor offices with two people being recorded from a crappy smartphone mic or desktop mic? (It could've been! But it's an important distinction.)
And there's more factors than that, but you get the picture (e.g. are they trying to "clean up" the transcript afterwards by feeding it to an LLM, are they attempting to pre-process audio before transcription also in an attempt to boost accuracy)
There's a lot of ways to do it, meaning, there's a lot of ways to screw it up.
It often feels like the AI industry is continually glossing over the fact that capability and reliability are fundamentally different qualities. We tend to use "accurate" and "reliable" interchangeably, but they describe different things. A model can ace a benchmark (capability/accuracy) and still be a liability in production (reliability).
Just look at recent reactions to yet another release from METR showing improved capabilities. But the less talked about part is how their measure is for a 50% success rate (and the even lesser talked about secondary measure they have at 80% success rate has a drastically lower time-horizon for tasks). https://metr.org/
I implement AI systems for enterprises and I don't know any that would ever be okay with 80% reliability (let alone 50%).
It just feels like for some reason this is all being relearned with LLMs. I guess shortcuts have always been tempting. And the idea of a "digital panacea" is too hard to resist.
Makes me think of how pre-chatGPT I still could barely handle most recipe blogs because of their well known attempts at "filling space". And yea the problem is significantly magnified now everywhere else.
Anyway, my point is, whether or not someone uses AI is almost secondary in a way (even though it can seem pretty obvious to most of us when it's being used). All that matters is if the writing seems like it cares more about throwing words at people instead of actually conveying its points in a way to elicit understanding.
I wouldn't under estimate switching friction.
I'll give a shoutout as well to Glimpse: https://github.com/LegendarySpy/Glimpse
The latest open source local STT models people are running on devices are significantly more robust (e.g. whisper models, parakeet models, etc.). So background noise, mumbling, and/or just not having a perfect audio environment doesn't trip up the SoTA models as much (all of them still do get tripped up).
I work in voice AI and am using these models (both proprietary and local open source) every day. Night and day different for me.
They could've released something like that years ago (the discovery service it's built on has existed for over a decade) but creating a simple, accessible, unified CLI for general integration apparently wasn't worth it until agents became the hot thing.
I wonder when / if there will be a rug pull on all of this. Because I really don't see what the long-term incentives are for incumbent tech platforms to make it easy for automated systems to essentially pull users away from the actual platform. I guess they're focused on the short term incentives. And once they decide the party's over, promising upstarts and competition can get absorbed and it'll be business as usual. Idk, we'll see.
> Chunks the audio when the model believes based on the words said by the user that they have completed their utterance.
Source: https://developers.openai.com/api/docs/guides/realtime-vad
OpenAI's Semantic mode is looking at the semantic meaning of the transcribed text to make an educated guess about where the user's end of utterance is.
According to Deepgram, Flux's end-of-turn detection is not just a semantic VAD (which inherently is a separate model from the STT model that's doing the transcribing). Deepgram describes Flux as:
> the same model that produces transcripts is also responsible for modeling conversational flow and turn detection.
[...]
> With complete semantic, acoustic, and full-turn context in a fused model, Flux is able to very accurately detect turn ends and avoid the premature interruptions common with traditional approaches.
Source: https://deepgram.com/learn/introducing-flux-conversational-s...
So according to them, end-of-turn detection isn't just based on semantic content of the transcript (which makes sense given the latency), but rather the the characteristics of the actual audio waveform itself as well.
Which Pipecat (open source voice AI orchestration platform) actually does as well seemingly with their smart-turn native turn detection model as well: https://github.com/pipecat-ai/smart-turn (minus the built-in transcription)
The cascading model (STT -> LLM -> TTS), is unlikely to go away anytime soon for a whole lot of reasons. A big one is observability. The people paying for voice agents are enterprises. Enterprises care about reliability and liability. The cascading model approach is much more amenable to specialization (rather than raw flexibility / generality) and auditability.
Organizations in regulated industries (e.g. healthcare, finance, education) need to be able to see what a voice agent "heard" before it tries to "act" on transcribed text, and same goes for seeing what LLM output text is going to be "said" before it's actually synthesized and played back.
Speech-to-Speech (end-to-end) models definitely have a place for more "narrative" use cases (think interviewing, conducting surveys / polls, etc.).
But from my experience from working with clients, they are clamoring for systems and orchestration that actually use some good ol' fashioned engineering and that don't solely rely on the latest-and-greatest SoTA ML models.
For anyone curious: https://flux.deepgram.com/
I think it's so easy for people to discount "mental energy" since culturally we don't often acknowledge it as a finite resource the same way we do physical energy. Well maybe the problem is we view them as separate things in the first place.
When I was younger I just didn't have to worry about so much stuff.
They have their Granite family of models, but they're small language models so surely significantly less resources are going into them.
I don't think it's going to be anything like 50% or even 30% of users using non-flagship hardware or software products. But it could still be significant. And I think the more important thing isn't going to be market share as much as proof of viability. More successful examples will beget more.
It's about planting seeds from which future digital ecosystems can grow -- that have interoperability, functionality, and openness built into their foundations.
I believe that what drove you to make this post and the way I feel is not unique and are part of a larger swell in similar sentiments.
You throw in other factors too like the mass tech layoffs and the continued doubling down of tech barons on their cravings to intermingle with the surveillance state and military industrial complex... I just can't see how the future doesn't have more people disillusioned with the current state of the tech industry.
I think big tech will continue to overplay their hand and the mess that comes after will be an opportunity to give people what they want and show alternatives to what's already been done that we know won't work out.