Hacker News Reader

Comment by toddmorey | Hacker News Reader

The conversation examples are impressive but this made me say holy shit out loud: "it runs locally on a MacBook Pro M4 Max or better. And we figured out how to train Dessa on only one NVIDIA GB200 NVL72."

If it really can replicate that low latency + that expressiveness on local devices, that's one heck of an accomplishment for "indie" AI.

View on HN

> that's one heck of an accomplishment for "indie" AI.

It's a nice accomplishment, but I think we have very different definitions of "indie".

https://en.wikipedia.org/wiki/Block,_Inc.#

> one NVIDIA GB200 NVL72

That’s around ~$4-6 million in hardware cost (one rack, 72 GPUs + accompanying hardware).

Many AI shops are funded well into tens of millions if not hundreds, for early rounds.

Also the team size is really small (< 10???). So yes, indie might be a stretch but still very impressive, relatively speaking.

Now taking the other side, they didn’t have to worry about hosting/billing/subscriptions or any of that yet. It’s a research demo at this point.

Also a speech to speech model it can't do any tool use or you can't guide it much am I mistaken? How will it be used for agentic uses is the part I don't get. Edit: this is response from Argon if anyone was also curious: You are not mistaken about Dessa: because it was trained solely on raw audio with zero text data, it cannot read text system prompts or output code for tool calls on its own. To use it as an agent, developers must either steer it purely through spoken voice instructions or pair it as a conversational "voice front-end" alongside a separate text LLM that listens to the audio and executes the tools in the background.

> No, but I speak the language of binary load lifters, which is very similar to your moisture vaporators

Perhaps boutique is a better term?

> > one NVIDIA GB200 NVL72 That’s around ~$4-6 million in hardware cost (one rack, 72 GPUs + accompanying hardware).

that’s a good clarification, skimming some of the comments here I was thinking they trained it on a single GB200 gpu. THAT would be a huge win for indie AI

Edit: Nvidia GB200 does not refer to a single gpu, Nvidia’s naming scheme is very confusing. Considering past gpus were called H100, H200 etc, I thought GB200 would just be a newer GPU, but it does actually refer to their 30+ gpu rack scale systems. The claim of “we trained this on a single rack of gpus” is definitely much less impressive than “we trained this on a single gpu”

> Nvidia GB200 does not refer to a single gpu, Nvidia’s naming scheme is very confusing. Considering past gpus were called H100, H200 etc, I thought GB200 would just be a newer GPU, but it does actually refer to their 30+ gpu rack scale systems.

It's even more confusing than that :)

    GB200 = 1 CPU + 2 B200 GPUs (packaged together)

    NV72 = A rack of 72 B200 GPUs (or 32 GB200 superchips)

Think of GB200 as something like Apple's 'Ultra' chip. Not exactly, but similar. They fuse the chips together with a very fast interconnect (connection).

Wait, seriously, a rack of 72 GPUs costs $4 million?

Hard to find concrete pricing [0], but apparently a pair can cost as much as $8.8M (depending on config)[1]

[0]: https://www.nvidia.com/en-us/data-center/gb200-nvl72/

[1]: https://www.tomshardware.com/tech-industry/artificial-intell...

This rack has 14 TB of ultra-fast VRAM.

> they didn’t have to worry about hosting/billing/subscriptions or any of that yet

Or marketing, which they kinda get for free. So much research goes unseen, there’s huge competition for attention, and most research is very limited in that regard

I would say that’s the biggest factor that makes the team “not indie”

At the same time, like you said, still amazing accomplishment, very cool model, and great work

oh, hacker news, where someone will act like a ten person shop with a seven million dollar computer might be indie

Speech is in quite a weird spot when it comes to locally running it. Some of the best STT and TTS models run at faster-than-realtime speed on decent consumer hardware with quite small models (so also no huge RAM requirements), but the tooling and library ecosystem is much more fragmented, making it a lot harder to build actual working products around it. There is also a much more noticeable coverage / quality gap when it comes to languages other than English.

As a speech-to-speech model Dessa is of course in a different ballpark, but given the intelligence it exhibits, it seems to be working with not that many parameters. The developers also seem to be admitting that to some degree ("in fairness, LLMs also have to learn C++ and whatever the hell Angular is"). I'm wondering what the utility of this would be. Given that it lacks both broad world knowledge and tool calling integration in any way, that seems kind of like a dead-end.

Probably Dessa started those demo conversations with some context, so maybe it could receive documents and speak about them. If those were random conversations started with no context, the people at Block probably recorded a lot of uninteresting chat and cherry picked the few good ones.

What prevents them from adding tool integration?

How would those tool calls work? Would it say what it wants and another model/system act on that and 'read aloud' the document to it?

I guess that's somewhat how we do it, so maybe it would work.

I suspect the non-tech-demo version would be multimodal, and include support for textual tool calls...

Converging on what AI companies are already doing... just from the other direction.

Current tool calling just parses llm output for flags that indicate a tool should be used.

How would you do that when output is raw audio buffers. You would need your conditionals to be like: if (audioDataArraySeemsLikeAnEmail(llmResponse)) {}

It's a bit strange to call it a dead end when this was trained on just a single NVIDIA GB200 NVL72. LLMs couldn't do much back then, either.

But the thing that makes this special, is explicitly cutting things out that make LLM's capable of that. As gpm's sibling comment points out, if you would add that back in, you would just end up in the converging to the same place (with the same downsides).

The only way around that would be to somehow get all world knowledge and tool calls into a audio space, which seems like quite the stretch.

> be to somehow get all world knowledge and tool calls into a audio space, which seems like quite the stretch.

It seems quite easy? They have the natural language bit nailed, they just need to convert the text with it. And it wouldn't be hard to extend architecture this into outputting tool calls.

This thing has the "the natural language nailed" to the same degree that a BERT has. That's a far cry from the intelligence level required to output text of any sensible length, and even farther from the structural adherence required for tool calls.

Highly-structured outputs (like what you'd need for tool calls) seems like it'd be easier to put out than arbitrary text, no? At the very least, even something like BERT could make the decision of “Would any of my available tools be useful here?” and delegate to some deeper-thinking model to structure that tool call.

I don't know whether people are just ignorant but it is heavily likely say the S2S without any intelligence can run on a pro-sumer device. Just because it is "OpenAI" and "Google" and not public knowledge or will be open source/public weights doesn't mean you can't reasonably infer.

> Given that it lacks both broad world knowledge and tool calling integration in any way, that seems kind of like a dead-end.

In its current form probably. I think the key to taking this further would be to split the listening model from the speaking model, and have some model(s) in between that can do those database/knowledgebase queries and tool calls and such and influence how the speaking model response to what the hearing model perceives. That's probably pretty similar to how ChatGPT Live and similar are operating (STT→LLM→TTS), but Dessa demonstrates that it doesn't have to necessarily be text/language, specifically, as the intermediary — and therefore it doesn't have to necessarily be an LLM, specifically, as that intermediate model.

How would you feed db/knowledge queries back to the model? Do TTS and try to stuff context with the synthesized audio?

It's speech to speech, but could another model interface with it in latent space somehow?

You could hook up a LLM to give it intelligence, if there were some interface

What STT and TTS models do you recommend that would run well on consumer hardware?

I use Kokoro 82M and it's good overall, but it does make mistakes on some common phrases, and it's not real time fast (runs at about 6-7X content-to-encoding time ratio on my home server)

>> I'm wondering what the utility of this would be.

Exactly, it does feel next level compared to STT -> LLM -> TTS setups, but unless you can steer it effectively to do something like helping a customer do a complex task it is just a chatbot?

I can't even imagine what things like RAG, system prompts and other context setups would look like with this.

> it runs locally on a MacBook Pro M4 Max

Open weights or GTFO

Jack said they will release the weights.

I mean cartesia did one of their first demos on a Mac book pro, live.

Still extremely impressive but not unheard of.

Reply on news.ycombinator.com