Hacker News Reader

Their preprint showed it worked well with small batch size, but most prod deployments right now have rather large batch sizes from serving multiple users

View on HN

That's a great answer, thanks. I'd missed that detail.

Honestly, a lot of those issues could be remedied with more optimized kernels, which LLMs could definitely write quite well now. I wouldn't be surprised if some deployments of Gemini Flash switched to diffusion for cases where there is low utilization, since Google showed you don't need to retrain an autoregressive model to rework it for diffusion inference.

> since Google showed you don't need to retrain an autoregressive model to rework it for diffusion inference

IIRC you do need to retrain the model, just closer to the level of finetuning rather than e.g. pretraining from scratch. Which also means that the model will perform differently with a different capability profile.

Thank you for the correction. You dont need to retrain from scratch

If that's true, that's the real answer, or more like anty-answer because you're saying basically: hey, nothing should stop them.

Google's gonna Google...

I wonder if it'd work better as a local model

Reply on news.ycombinator.com