Hacker News Reader

Damnit, I've been waiting on an open weights model like this one, still no dice :(

diffusion gemma?

Was mostly an experiment. You're not going to use it for any real coding. Hopefully we'll see another updated diffusion gemma in the future.

It pretty bad. Just ask it to quote something like the first paragraph of moby dick (i.e. "call me ishmael...") it barely gets past 5 words before it completely gets it wrong. Even the worst normal 4B models get 3-5 sentences.

That’s because diffusion models struggle with memorizing sequences while it’s essentially what autoregressive models excel at.

I’ve tried to fine tune a diffusion model to memorize an MD5 hash and after days of hacking on it, it failed to completely memorize it. Autoregressive models of any size learn it over <1000 iterations.

Reply on news.ycombinator.com