50,078 karma · joined July 3, 2007
Playing with fun tech.
Mastodon: @mmastrac@hachyderm.io (https://hachyderm.io/@mmastrac)
Email me: matthew@mastracci.com
LinkedIn: https://www.linkedin.com/in/matthewmastracci/I only learned this the hard way reimplementing diffusiongemma. I had ideas on how to fix it but no cluster to train and experiment, hah.
We're nowhere near a Fable-class model IMO, but things are going to get interesting in this next year.
For example, I have a small posix-shell-based LLM harness that can SSH into my NAS and run organization tasks using the local DS4Flash that I have right now. It's already been a massive help for me to keep me organized, and that's just 2x DGX Spark's worth of compute.
They were briefly on sale with a $200-off coupon, but they show up on warehouse deals from time-to-time as well.
I decided to take the plunge and get myself four sparks at a decent price (and bought the QSFP cables from AliExpress because they are literally 1/2 the price of Amazon), even knowing Apple was going to release new hardware and there's probably a spark 2 on the horizon. It looks like this is going to be a decent fit for what I need. I've been experimenting with a two-node DS4 and it's _good_ at some tasks, but it really just spins its wheels when it hits the limit of what it can reason through.
I can offload mundane/basic tasks to DS4 on two sparks, but I've been pushing it harder on some novel work and it just can't run on its own at all beyond a certain complexity level.
I would love to see an Opus-4.8-level local model but TBH I just haven't got there yet. The models I've tried so far _are_ good but they aren't able to solve tough technical challenges, regardless of harness/prompting/etc.
Crater runs are probably one of the most valuable tools that Rust has.
Comment blocks are <= 7 words, function names <= 4 words. User-facing message strings should be <= 10 words. Use an active voice, no stage performances, and pick the most common word when choosing among alternatives.
Limiting the number of words is the strongest factor in cleaning up the output, IMO.
For older code I've instructed it to delete all the comments, and then I re-comment it using a new session and these guidelines, asking it to rejustify the need for every comment to itself.
I'd be curious to see your CUDA approach - happy to combine efforts as well.
I like the model a lot and it's fairly good at reasoning. You can also really bend it to your needs. It's designed for machines with more compute than memory bandwidth but IMO does really well on metal.
I've got it up to ~15tok/s on M3-class machines, but I wager there's a bunch of perf on M5 that I just don't have hardware access to unlock.
I tried to implement MTP using the other Gemma MTP heads but I failed to move that perf needle. There's some interesting research to be done about pre-seeding the diffusion canvas from draft models. DiffusionGemma with the right drafter can hit 20-30 tok/s on my machine, but I've been unable to combine the two together to make it faster than what it's been running at so far.
Please, though, take a pass at humanizing the text on the page. It's Clauded up all over and makes it hard to read.
I've got a number of tiled-load kernels, and a top-k attention kernel that you might find interesting.
What I also learned is that MLX/vLLM is probably within ~20% or so of the absolute max perf on Mac. I found some improvements over what they were doing, but we're at the point where it's challenging to optimize without per-stepping kernels.
I found a few improvements over stock DiffusionGemma along the way, like using top-k attention, which drastically improves perf on my mac without sacrificing any of the benchmarks I was able to throw at it.
FWIW some of the issues with Gemma being slow on Mac are specific choices they've made in the architecture that make it challenging to make use various optimizations that have popped up recently. I think a Kimi K3-style network hybrid with the diffusion bits of DiffusionGemma could have some serious sway.
I think that diffusion still has an edge locally, but with some architecture tweaks and CPU improvements it would actually be a winner (ie: training the network for smaller token batch sizes or flexibility in attention heads, a less expensive attention mechanism, and others).