Hacker News Reader

Comment by ttul | Hacker News Reader

I suspect that Dots and Duet are very high margin drivers for the labs. Personal assistant chat sessions don’t have to be realtime, so workloads can be scheduled on low priority batches and be waited if higher value work comes into a queue.

Inference providers are always trying to fill inference batches and having a stream of low priority requests can help pack batches that might have gone empty, without raising the latency of higher priority requests such as live chat sessions.

While Dot and Duet certainly need to be useful to work out as products, I think the gross margin mechanics are just as interesting.

View on HN

> workloads can be scheduled on low priority batches

I feel like there's enormous potential still uncovered in the coding agent space for such highly-discounted "spot" style capacity. Give me a way to have agents continuously crawl over my codebase looking for bugs, come up with test cases that prove the bug, open a PR. I don't care if it takes 12 hours to find something that would take 5 minutes if prompted with "on-demand" / "high-priority" requests, because I'm not prompting for it directly, I'm not aware of how much time it takes, all I see is continuous improvement.

Isn’t the KV cache a huge component of the overall costs, and wouldn’t this make this problematic? Cache writes are an expensive part of LLM tokens, even more so for caches that have a long expiration time.

Cache is a huge component of "high priority" costs because it essentially forces the GPU to be idle (holding earlier prompt text in GPU memory) while waiting for newer input tokens. When an agentic session is active, this is of course much more preferable compared to loading in hundreds of thousands of input tokens again and again on different GPUs, but "walking away from the laptop" while waiting for the cache to expire is something that royally sucks today for inference providers. (Caveat: this is my layperson's understanding, I don't work in the space, I'm likely not up to date on best-practice caching practices)

"Low-priority" inference tasks like that, though, would never need to be cached. Again, who cares if it takes 12 hours? If you didn't initiate the prompt, then what do you care? You just randomly get notified when something is already ready for you to take a look at.

Isn’t caching much more nuanced, with different tiers (DRAM, local disk, remote disk, etc), and isn’t filling these caches something that happens asynchronously (ie while the GPU is still working on the previous batch) ?

I think reality is much more nuanced than what you’re sketching.

But that's my point - throw the "background code review" stuff out of cache, or into object storage, or whatever, just something to draw upon to fill inference batches opportunistically.

I mean, at the end of the day, either inference providers are having difficulty filling batches, in which case there's opportunity on the table to somehow fill the batches with discounted, time-insensitive, "background review" style work, or they don't have that difficulty, in which case it's a moot point. Exact cache architectures are beside the point.

It might become a triple-whammy benefit for them. More efficient compute, as you said, but also a bot capability creates a better moat around their infra ecosystem and the amount of training data that can come from a bot ecosystem is significant.

Models are becoming increasingly interchangeable but setting up and maintaining (!!) the ecosystem still takes expertise. Not everyone wants to host their own MicroVM infra and would happily pay Anthropic/Grok/OpenAI to do it for them.

I don't know. I think the moat will come from enterprise deals and deep enterprise integration, like it does for almost all software products so far.

I think for B2C products to have moat you need some type of network like social media apps. Or can you think of any other examples of B2C software products that have a strong moat?

I like how you can exchange the word moat with monopoly.

yeah moats are bad for everyone who isn't a shareholder or employee so i don't get the concern for whether or not there's a moat, its like watching shark tank

Agree

Lol please stop with this 'moats' discussion.

Moats become known ex-post not ex-ante. shake my head its getting boring! Every other post seems to be a moats and most of you haven't got a faintest clue of what a durable competitive advantage is and how it actually gets created.

“Claude, please port everything over from Dots to Duet. Make no mistakes.”

Haha, legit.

Personally I am someone with all the free time in the world guess what? I barely use LLMs! Im fortunate to have the ability to think critically for myself and ponder what exactly is the value-add of these tools. Look I have no doubt they provide solutions to problems at a tap of a button with some degree of variance w.r.t correctness and quality but people should take a step back and have a more nuanced thought process.

Its only the frontier labs that don't want you to do that and want to focus on 'go go go'. I'm seeing way too much frantic behaviour and people getting all caught up in it. What's panicking gonna do? Just relax and observe cautiously.

TBH I was thinking moat in terms of enterprise customers so I think we're actually on the same page. The majority of (non-tech) companies do not want infrastructure-they want capabilities. If anthropic can drive adoption for agentic capabilities in enterprise _running on their anthropic's infra_, that can help keep anthropic customers inside their ecosystem as there would be switching costs.

For the B2C side, my gut is telling me that Anthropic couldn't care less about that market, at least for now, with their bot product.

I just watched a YouTube video from How I AI[1] doing a bit of a preview on Duet. Their point was quite the opposite. The Duet agent is so slow that it's not actually particularly pleasant to use, even if it is smart. I tend to agree, and think speed is a key feature here. If it's not quick, it risks feeling disconnected.

I have, against my better wishes, rather given in to Grokbot over the last couple of weeks. It does feel very nice to use for assistant type tasks. At least part of that is because it feels pretty responsive.

[1]: https://www.youtube.com/watch?v=HYb5w4PfPZM

Grok can be fast because X.ai does not worry about gross margins yet. They can tuck those nasty secrets away deep inside SpaceX…

And there's also a lock in mechanism. With regular chat, you can simply switch to another provider, and with a harness or API, switching is pretty easy too but here, it's more difficult.

I think these are actually very expensive, particularly for Muse giving free access to the masses… These agentic workflows are just context burners

Reply on news.ycombinator.com