Hacker News Reader

Comment by hobofan | Hacker News Reader

Speech is in quite a weird spot when it comes to locally running it. Some of the best STT and TTS models run at faster-than-realtime speed on decent consumer hardware with quite small models (so also no huge RAM requirements), but the tooling and library ecosystem is much more fragmented, making it a lot harder to build actual working products around it. There is also a much more noticeable coverage / quality gap when it comes to languages other than English.

As a speech-to-speech model Dessa is of course in a different ballpark, but given the intelligence it exhibits, it seems to be working with not that many parameters. The developers also seem to be admitting that to some degree ("in fairness, LLMs also have to learn C++ and whatever the hell Angular is"). I'm wondering what the utility of this would be. Given that it lacks both broad world knowledge and tool calling integration in any way, that seems kind of like a dead-end.

View on HN

It's a bit strange to call it a dead end when this was trained on just a single NVIDIA GB200 NVL72. LLMs couldn't do much back then, either.

But the thing that makes this special, is explicitly cutting things out that make LLM's capable of that. As gpm's sibling comment points out, if you would add that back in, you would just end up in the converging to the same place (with the same downsides).

The only way around that would be to somehow get all world knowledge and tool calls into a audio space, which seems like quite the stretch.

> be to somehow get all world knowledge and tool calls into a audio space, which seems like quite the stretch.

It seems quite easy? They have the natural language bit nailed, they just need to convert the text with it. And it wouldn't be hard to extend architecture this into outputting tool calls.

This thing has the "the natural language nailed" to the same degree that a BERT has. That's a far cry from the intelligence level required to output text of any sensible length, and even farther from the structural adherence required for tool calls.

Highly-structured outputs (like what you'd need for tool calls) seems like it'd be easier to put out than arbitrary text, no? At the very least, even something like BERT could make the decision of “Would any of my available tools be useful here?” and delegate to some deeper-thinking model to structure that tool call.

> Given that it lacks both broad world knowledge and tool calling integration in any way, that seems kind of like a dead-end.

In its current form probably. I think the key to taking this further would be to split the listening model from the speaking model, and have some model(s) in between that can do those database/knowledgebase queries and tool calls and such and influence how the speaking model response to what the hearing model perceives. That's probably pretty similar to how ChatGPT Live and similar are operating (STT→LLM→TTS), but Dessa demonstrates that it doesn't have to necessarily be text/language, specifically, as the intermediary — and therefore it doesn't have to necessarily be an LLM, specifically, as that intermediate model.

How would you feed db/knowledge queries back to the model? Do TTS and try to stuff context with the synthesized audio?

>> I'm wondering what the utility of this would be.

Exactly, it does feel next level compared to STT -> LLM -> TTS setups, but unless you can steer it effectively to do something like helping a customer do a complex task it is just a chatbot?

I can't even imagine what things like RAG, system prompts and other context setups would look like with this.

I don't know whether people are just ignorant but it is heavily likely say the S2S without any intelligence can run on a pro-sumer device. Just because it is "OpenAI" and "Google" and not public knowledge or will be open source/public weights doesn't mean you can't reasonably infer.

Probably Dessa started those demo conversations with some context, so maybe it could receive documents and speak about them. If those were random conversations started with no context, the people at Block probably recorded a lot of uninteresting chat and cherry picked the few good ones.

What prevents them from adding tool integration?

Current tool calling just parses llm output for flags that indicate a tool should be used.

How would you do that when output is raw audio buffers. You would need your conditionals to be like: if (audioDataArraySeemsLikeAnEmail(llmResponse)) {}

How would those tool calls work? Would it say what it wants and another model/system act on that and 'read aloud' the document to it?

I guess that's somewhat how we do it, so maybe it would work.

I suspect the non-tech-demo version would be multimodal, and include support for textual tool calls...

Converging on what AI companies are already doing... just from the other direction.

What STT and TTS models do you recommend that would run well on consumer hardware?

I use Kokoro 82M and it's good overall, but it does make mistakes on some common phrases, and it's not real time fast (runs at about 6-7X content-to-encoding time ratio on my home server)

It's speech to speech, but could another model interface with it in latent space somehow?

You could hook up a LLM to give it intelligence, if there were some interface

Reply on news.ycombinator.com