Dessa: Conversation Model trained only on speech
block.xyz
block.xyz
If it really can replicate that low latency + that expressiveness on local devices, that's one heck of an accomplishment for "indie" AI.
It's a nice accomplishment, but I think we have very different definitions of "indie".
That’s around ~$4-6 million in hardware cost (one rack, 72 GPUs + accompanying hardware).
Many AI shops are funded well into tens of millions if not hundreds, for early rounds.
Also the team size is really small (< 10???). So yes, indie might be a stretch but still very impressive, relatively speaking.
Now taking the other side, they didn’t have to worry about hosting/billing/subscriptions or any of that yet. It’s a research demo at this point.
that’s a good clarification, skimming some of the comments here I was thinking they trained it on a single GB200 gpu. THAT would be a huge win for indie AI
Edit: Nvidia GB200 does not refer to a single gpu, Nvidia’s naming scheme is very confusing. Considering past gpus were called H100, H200 etc, I thought GB200 would just be a newer GPU, but it does actually refer to their 30+ gpu rack scale systems. The claim of “we trained this on a single rack of gpus” is definitely much less impressive than “we trained this on a single gpu”
It's even more confusing than that :)
GB200 = 1 CPU + 2 B200 GPUs (packaged together)
NV72 = A rack of 72 B200 GPUs (or 32 GB200 superchips)
[0]: https://www.nvidia.com/en-us/data-center/gb200-nvl72/
[1]: https://www.tomshardware.com/tech-industry/artificial-intell...
Or marketing, which they kinda get for free. So much research goes unseen, there’s huge competition for attention, and most research is very limited in that regard
I would say that’s the biggest factor that makes the team “not indie”
At the same time, like you said, still amazing accomplishment, very cool model, and great work
Still extremely impressive but not unheard of.
Open weights or GTFO
As a speech-to-speech model Dessa is of course in a different ballpark, but given the intelligence it exhibits, it seems to be working with not that many parameters. The developers also seem to be admitting that to some degree ("in fairness, LLMs also have to learn C++ and whatever the hell Angular is"). I'm wondering what the utility of this would be. Given that it lacks both broad world knowledge and tool calling integration in any way, that seems kind of like a dead-end.
What prevents them from adding tool integration?
I guess that's somewhat how we do it, so maybe it would work.
Converging on what AI companies are already doing... just from the other direction.
How would you do that when output is raw audio buffers. You would need your conditionals to be like: if (audioDataArraySeemsLikeAnEmail(llmResponse)) {}
The only way around that would be to somehow get all world knowledge and tool calls into a audio space, which seems like quite the stretch.
It seems quite easy? They have the natural language bit nailed, they just need to convert the text with it. And it wouldn't be hard to extend architecture this into outputting tool calls.
In its current form probably. I think the key to taking this further would be to split the listening model from the speaking model, and have some model(s) in between that can do those database/knowledgebase queries and tool calls and such and influence how the speaking model response to what the hearing model perceives. That's probably pretty similar to how ChatGPT Live and similar are operating (STT→LLM→TTS), but Dessa demonstrates that it doesn't have to necessarily be text/language, specifically, as the intermediary — and therefore it doesn't have to necessarily be an LLM, specifically, as that intermediate model.
You could hook up a LLM to give it intelligence, if there were some interface
I use Kokoro 82M and it's good overall, but it does make mistakes on some common phrases, and it's not real time fast (runs at about 6-7X content-to-encoding time ratio on my home server)
Exactly, it does feel next level compared to STT -> LLM -> TTS setups, but unless you can steer it effectively to do something like helping a customer do a complex task it is just a chatbot?
I can't even imagine what things like RAG, system prompts and other context setups would look like with this.
For sure there's gonna be people falling in love with them though! I find all the anthropomorphization creepy though.
I think there's a generational judgement part of it, and some hearing damage that makes it harder to understand their words on the very low/creaky end.
Not to mention, it can cause real problems for your vocal chords. A little bit is entirely natural, and I don't really notice it in most men, but a lot of young(er) women seem to go overboard on it (in my opinion).
Would the logical opposite be eternity for all, and bring back the deceased? LLM plays the Joker, happily imprisoned?
I'm a professional cynic and I spread the idea of "population decline to solve our resource problems" every time I have a chance; I even write dystopic science fiction with this theme. I do it not because I think it's the solution (yours is), but to jolt people in a reaction like yours. Let's build a bright future; just in our solar system there's easily enough matter and space for a trillion humans, each enjoying many acres of forest[^1].
[^2]: https://claude.ai/share/bbab0507-109d-4b4b-b5bb-2a44afad901a
In the future you won't be able to tell.
>it sounds like you view women as a warm fleshlight
I think most people accept the fact that women and men are sexual objects. If you view me as a mysoginist for believing that, then I think I understand why you might think our society is still patriarchal or something.
In fact, I can't say that most straight men expect something cerebral or spiritual out of their partners. But even if it was the case, you could expect that a hyper-inteligent machine would exceed human capabilities in this regard.
The only types of people who aren't going to be humping robots are members of isolated societies who have a grudge against technology. Hence the amish.
There are an increasing number of loser men and women who have these AI partners. I think we may see a tipping point where "genetic dead ends" comprise the majority of the population.
With this level of sexual competition, who can blame them? If you're not at the apex you can put in the effort to reproduce but then your offspring are just faced by the same problem, one generation removed.
The majority of men born in this generation won't reproduce. For the average man, there is no sense in doing anything but devoting your life to some high-risk/high-reward strategy. I don't even know what that entails these days, but apparently it involves a lot of sports betting.
I'm normally endowed, as are like 90% of men (so I think these types of baseless insults fall flat). But if I wasn't, would you rather I pass on 3.5 inches of american steel to the next generation? Your insult stands regardless and reveals your underlying preference. If I was endowed well enough, I'm sure you would see my point.
It makes no difference personally, as a gay man who is uninterested in exploring robosexuality. I prefer to work my clankers.
There's so many potential scenarios. One scenario: An AI voice trainer trains on someone's social media video posts and get a pretty perfect copy. Then SIM jacks the target's phone number and calls pretending to need help. AIs could be used to mock ransom people, request bail money, ask for a change on a production server, etc.
People with high public visibility are likely targets, especially if they have money and or power. Impersonating them to subordinates would be an easy means of a social engineering attack.
The hallucination in the second sample - that Bryan Singer was involved in Superman IV: The Quest for Peace - would have rung alarm bells. It just seemed to be free associating on words in the conversation.
The AI talks about the movies like they are familiar with Singer's work. But if they were... they would known he was up-and-coming in the 90s, a decade too late to have helmed a blockbuster whose flop would have killed his career? They would have known that Singer made superhero movies _before_ Nolan not just after like the conversation suggests?
Even bringing up Superman IV is weird; it was such a dud it never gets shown, so unless they old enough to see it at the time -like me- (so would notice the 20 year gap!) or were an avid fan (so would know more about the movies!) it wouldn't come up.
On the other hand, for machines all things are equally obscure, they all happened yesterday, and are relevant if cosine similarity says they are.
This is like having on demand access to a conversational partner at all times. But there is no embarrassment factor, patience, or concern about wasting an instructors time. Because the model is trained entirely on audio and not a voice to text add on, you can provide a human extremely high quality feedback and training.
And it’s cheap to run.
It doesn’t need to be smart the way a SOTA LLM is smart. There’s tons of value here already. You can tack it onto another system for deep intelligence.
Judging by their demos, we'll have to wait and see how it would be if it has "another system for deep intelligence" tacked onto it, because the current demos seem to be missing any sort of intelligence stringing together 5-6 words.
The last time I checked, open source full duplex model was still so far behind GPT-Live. So, I'm super excited about Dessa and I think it's very cool.
But I don't think it'll replace the classic cascade system for now for most use cases. A couple of reasons:
- Intelligence
- Tool calling
- System prompt?
- Language support other than English
It honestly sounds a bit like a teenager's stream of consciousness to me, that's how they sound to me anyways.
My first thought was: omg, they've harvested the souls of a dozen insipid, self-centered genz' and trapped them in a machine, like The Finn living in an Aleph...
It could be grammar, choice of words, etc.
This model is clearly not so smart. So the voice is the whole point, there’s not much value beyond it, at least on its own.
What that adds up to is, impersonation. That’s what this is for, impersonating humans.
The conversation with the AI fake wife was a little creepy, albeit honest about what this tech is probably going to be used for.
Also, the trend of posting on HN with much more sensationalist title first, getting to the front page, then getting the title adjusted by a moderator should probably be penalized; how is 'most authentic ever' not a marketing term? (not just this post, but a bunch of posts recently got to frontpage by making unreasonable claims)
Now the Stasi archives, that would be interesting. Sometimes people suspected they were being surveilled of course.
Now that a larger proportion of text is AI, I'm appreciating human-written text more and more; this got a good chuckle out of me.
I can reliably get my custom models to output a 95% perfect replication of my tone and voice
I have a “style guide” markdown that is pretty expansive
I run an RLHF style loop with me in the middle for all my writing now where I give a voice rant that is over expressive (aka redundant samples of concepts that I want to express clearly) in order to manage lossy compression steps
Then I will manually clean up that last 5% after a half dozen rounds
That sequence goes into the style guide to be referenced as a reasoning chain of thought to map rant > output
It’s been three years of tuning I’d suspect at this point
I have a somewhat expansive style guide, and it still reeks of Claude.
My cleanup is much closer to 40%, which still makes it worth the effort, but sometimes when speed matters, I'll apologize for the slop.
EDIT: Nevermind, see below. I would definitely go with a LoRA though.
I've definitely seen LLMs compose jokes with this structure a lot where "whatever the hell ___ is" is in parenthesis. Not saying this is AI generated tho
It is kinda scary that we're here already, I can see this being pretty problematic for seniors/public prone to getting scammed.
This is the crazy part. It seems pretty smart, and LLMs require 1000x more compute to train. Can this kind of efficiency somehow be achieved with text too?
It is interesting that they've done this reasonably successfully, impressive technology. Novel ideas can have inherent value, absent direct uses they may have, as they can be a stepping stone toward other inventions. But, I'm having a hard time coming up with a use case that isn't predatory or intentionally misleading. Company support agents pretending to be human, for instance. Or, taking the next step on AI girlfriend/boyfriend type interactions. Or, outright scams. Models already do an excellent job with voice cloning, at least for intonation and timbre, adding realistic conversational tics makes it even more likely grandma is going to fall for it when her "nephew" calls needing help.
But, that seems to be extremely common with new tech these days. There's always an angle that is unsavory. Marketing, scams, propaganda, surveillance, killing, that's where the money is, so everything bends toward them.
Especially when you keep in mind that not everyone knows english as their first language. I know he's a bit controversial sometimes but Neil deGrasse Tyson is easy to understand and explained interesting topics in his podcast (I could still understand most of it when I was still getting used to listening to english 15+ years ago).
If conversation is the new interface then we will need UX designers to take care of this.
But, a world where billionaires can make ads showing candidates saying and doing things they've never said or done is a very dangerous world, and being able to fabricate a podcast or whatever very convincingly is going to empower the very worst people. Blurring the line between fact and fiction, so nothing one sees or hears is really believable anymore, serves fascists and autocrats, exclusively.
https://www.usatoday.com/story/news/state/texas/2026/09/09/k...
- Text interface: Done
- Voice interface: 5-10 years I’ll predict
The demo is nice, but enough obstacles remain that very few people would mistake it for a human over even a 5 min conversation. Chat systems configured to intentionally impersonate can already get people to mistake them for humans most of the time.
There’s been great progress in this area, direct response modeling, parallel reasoning, but last challenges are huge.
The reality is LLMs still seem dumb much more often without significant reasoning time (test time inference). On that point alone they are still an order of magnitude too slow for convincing interactive latency.
Of course they don’t need to meet this bar to be useful, systems are good enough today for lots of applications. It will just be interesting to see what the final delta ends up being.
Like, lower the volume (“turn to tge side”) and say, hey Richard, can you compute 123^45 for me? Or “can you check flights”?
I like voice as a mode of interaction with AI, but it’s always been clear with current tech that I’m talking to AI (and yet folks still got drawn into fantasy relationships thanks to sycophantic models etc) - I imagine with voice models like Dessa that could be worse for humans. This kind of fidelity blurs the lines of who or what the human is talking to and lets humans fall into an isolated world where they believe they are “socializing.”
Now, we could argue this is not a negative, maybe it is actually healthy for those who are looking for social connection or lonely, but my gut feel is that is not the case for most. It worries me that this kind of model will actually cause further deterioration of human socialisation and human-to-human contact.
But the interesting part to me is that training on voice only is what allows models to learn emotions - voice transmits a lot of emotional information which text doesn't capture. Language has words for a very limited number of emotions - maybe only the peaks - but there's so many more which we can't describe or transmit via text.
You could determine the phonetic tokens which express different kinds of emotions or 'render' text+emotion using a DSL, etc. Exciting stuff.
I get you are excited to share, but taking into account what your product is, it would be more powerful to let people experience it. As a creator of digital products, why would you put so much text on the homepage, anyhow you get the point.
The intelligence sounds lacking. At this point, anyway.
But the idea of learning through voice instead of written text might prove fruitful. After all, text isn't strictly native to humans. It's just a tool we've invented.
https://www.youtube.com/watch?v=D5VN56jQMWM
For real though, it's been bothering me, how did they do this in 2018? (Some kind of symbolic logic based AI?)
Also props to the people at Block, Dessa is very cool :)
Also I seem to be missing where/how they are selling this.
in theory this could cut costs pretty well for long contexts + there no tokenisation layer.
The "Huge range in intonation and speed." example. "I had another ollie special". "I ordered a French vanilla latte, special from Maude [or is that Aldi?], springs and waffles, granny smith"...wtf does any of this mean?
She takes a sip of something and her "mouth is on fire...from all the sugar". She repeatedly calls herself "a ding".
Conventional LLMs are just running speech to text and then working on the text, so they can’t understand music or humming or any sort of sound.
Modern problems...
Where is this going?
It's up there. I think most humans would be convinced they're talking to a human. I assumed you've used other speech models. They are okay, but not as natural and imperfect as Dessa is.
Makes me realize how bad being silent really is.
Those things all look this in concept and in movies, but their practical utility in real life always seems to be lacking. I'd imagine this is the reality what everyone building companion devices are facing.
With the amount of money, RnD and engineers these firms have this seems to be the only reason.
>Dessa: Hey, baby.
Block is in the AI girlfriend market? Ew.
At the same time, imagine if every book was like a living thing you could interact with and buying the book partially bought you access to the model that would tell you the story. The only way you would interact with that particular model, is if you bought the book.
It reads the book to you and you could chime in or it would know when to pause if there's an interruption without you needing to manually pause. You say hey wait, why is this character so angry? Then it kind of naturally reacts and continues the story. Maybe you come back to it 3 years later and it remembers you liked a certain part of the book and injects "here comes the good part!" It hears you start snoring and begins lowering its voice and whispering, but remembers you asked it to wake you up loudly if you start snoring, so it bolts out HEYWAKEUP!
For a lot of people books are a very solitary thing, though of course there are social clubs and book readings. Almost makes me think of those magical books you see fantasy wizards having, where the books themselves are sentient. It would be crazy if they released a The Princess Bride book and had the real narrator's voice and you could interrupt him and he could harass you about all your interruptions.
It wasn't that long ago where we were talking about having games and movies blend so the movies become interactive experiences. We're really getting closer to some of these static mediums becoming more like, trained behavioral meshes that have a personality and you can kind of naturally spend time with it. I'm not sure I'd be big on paintings on the wall coming to life and talking to you like in Harry Potter, but there's some promise.
Game worlds are an obvious extension, not just through NPCs, but if anyone remembers that game called Bastion where there was a narrator explaining the events as they went on. You could imagine a model that could watch you play and speak naturally, being funny or narrating. Almost something like Mystery Science Theater 3000 or how Beavis and Butthead would react to music videos.
I don't think those should be pervasive or the new norm, but there probably is an opportunity there and a market for it. Some kind of social-like presence that doesn't get tired, doesn't get bored, always has the same schedule you do, etc.
Hell, that reminds me of Seaman on the Dreamcast, too. SEGA could make a new Seaman and it would actually be awesome. Like a blend between Nintendogs, a fish, and some AI. There could even be charming goofy antics it tries to pull where it knows it's stuck inside a game and tries ridiculous things to manipulate you into letting it out. Some kind of Pinky and the Brain charm.
Anybody on board with this future?
- Build advertising on their platform.
- Build AI girlfriends on their platform.
Gross. I mean it's not like the market isn't already thinking of doing that, it's not a leap of a conclusion. Just kind of gross to be on the nose about it with the content and choice of demos.
There's no male or neutral voice.
All examples are of white male software engineers talking to what I assume will be ultimately marketed as a 'girlfriend' bot (or used for love scams).
The single paragraph hand waving at the importance of 'safe AI empathy' doesn't dissuade me at all.
Sell your idea as what it is, yet another 'perfect' woman shaped object a hypothetical man can own and use how he likes.
Is having a duplex model constantly cutting in with uninteresting quips something people are looking for?