HNHacker News
TopNewBestAskShowJobs

mmastrac

50,078 karma · joined July 3, 2007

Matt Mastracci

Playing with fun tech.

  Mastodon: @mmastrac@hachyderm.io (https://hachyderm.io/@mmastrac)
  Email me: matthew@mastracci.com
  LinkedIn: https://www.linkedin.com/in/matthewmastracci/
submissionscomments
mmastrac··on Muse Spark 1.3
Any idea what size this is?
mmastrac··on SK Hynix CEO sees memory chip shortage lasting until 2030
Do you have a source for more info? That seems odd to me - LLMs are particularly resilient to bitflips, so if anything the demand would pull on DDR5 in general, not on ECC specifically.
mmastrac··on How to build a diffusion language model
What's missing from these discussions is the real weakness of diffusion models: you can have two positions where two tokens need to be coordinated, but both spots are teetering between two valid states. They don't always collapse to the matching pair.

I only learned this the hard way reimplementing diffusiongemma. I had ideas on how to fix it but no cluster to train and experiment, hah.

mmastrac··on A 12TB Steam “teraleak” spills more than a decade of lost PC gaming history
Is there a lookup that gives you a title to blob mapping?
mmastrac··on Why open source rocks – a new SM750 (Silicon Motion GPU) HDMI Driver
Oh nice. I have a little PCIe/VGA one that I use for truenas and it's a bit flaky but I love that it's basically free from a PCIe perspective (just toss it off _any_ root). These are super useful and there are even m.2 variants.
mmastrac··on Our decision on Cursor following its acquisition by SpaceX
If you can explain how to make compaction not suck in opencode or Kilo Code, I would love to know how. I am really trying to make use of other harnesses but that's a major sticking point.
mmastrac··on GLM-5.3 is now open-weight
Yes, sorry 5.3 flash.
mmastrac··on GLM-5.3 is now open-weight
NVFP4 ~20-30tps (MTP + vision, no dflash2).
mmastrac··on GLM-5.3 is now open-weight
I previously posted that DS4Flash was _good_ but not _great_ on two DGX Sparks, but I have to say that GLM-5.3 is pretty amazing. It's been able to tackle all the random hard problems I've thrown at it and it has the intuition that DS4Flash seems to lack.

We're nowhere near a Fable-class model IMO, but things are going to get interesting in this next year.

mmastrac··on GLM-5.3-Flash
Sparks + cables + 10g SFP+ came out to ~$21,500 CAD
mmastrac··on GLM-5.3-Flash
I think I need one now that I have four - reduce is ring-oriented and still works I believe, but IIRC you only get 200gbps if you use _one_ of two connectx ports.
mmastrac··on GLM-5.3-Flash
What harness? I've had similar results as _Implicated_ said above - it's not done well in any of the tests I've tried with it. I currently have it hung off DS4Flash as a pseudo-vision tool and subagent only because of this.
mmastrac··on GLM-5.3-Flash
The cables are ~USD $50 from AliExpress although I'm not sure what the tariff situation is for Americans (I think I paid $75 all-in CAD for them)
mmastrac··on GLM-5.3-Flash
I already have a recycled QNAP-now-TrueNAS that has 10G connections into the fleet so the 1TB doesn't bother me at all. I did some rough math and I don't think I'll ever need to load weights off NFS for what I'm doing so far, but the capability is there.
mmastrac··on GLM-5.3-Flash
For me it's entirely because I have a bunch of projects with my own personal data that would be tough to do with openrouter/claude or any other cloud.

For example, I have a small posix-shell-based LLM harness that can SSH into my NAS and run organization tasks using the local DS4Flash that I have right now. It's already been a massive help for me to keep me organized, and that's just 2x DGX Spark's worth of compute.

mmastrac··on GLM-5.3-Flash
I bought 4x Asus GX10 with the 1TB option. I don't understand why, but it's the only model in the whole lineup that isn't priced insanely.

They were briefly on sale with a $200-off coupon, but they show up on warehouse deals from time-to-time as well.

mmastrac··on GLM-5.3-Flash
Weights on HF here: https://huggingface.co/zai-org/GLM-5.3-Flash

I decided to take the plunge and get myself four sparks at a decent price (and bought the QSFP cables from AliExpress because they are literally 1/2 the price of Amazon), even knowing Apple was going to release new hardware and there's probably a spark 2 on the horizon. It looks like this is going to be a decent fit for what I need. I've been experimenting with a two-node DS4 and it's _good_ at some tasks, but it really just spins its wheels when it hits the limit of what it can reason through.

I can offload mundane/basic tasks to DS4 on two sparks, but I've been pushing it harder on some novel work and it just can't run on its own at all beyond a certain complexity level.

I would love to see an Opus-4.8-level local model but TBH I just haven't got there yet. The models I've tried so far _are_ good but they aren't able to solve tough technical challenges, regardless of harness/prompting/etc.

mmastrac··on IPFS Maintainers Winding Down
I'm currently on sabbatical looking for volunteer work. I always thought IPFS was an interesting project, might be happy to pitch in some of my now-free bandwidth.
mmastrac··on Rust 1.98 got a P-critical miscompilation
Wild how infrequently this happens. There have only been a handful of compiler crashes I've been hit by (outside of nighties), and I don't recall the last one that has happened since near the 1.0 days.

Crater runs are probably one of the most valuable tools that Rust has.

mmastrac··on Claudette: Make Claude stop talking like a BuzzFeed article
I've started giving these instructions and I think I've been much more successful in generating clear output:

Comment blocks are <= 7 words, function names <= 4 words. User-facing message strings should be <= 10 words. Use an active voice, no stage performances, and pick the most common word when choosing among alternatives.

Limiting the number of words is the strongest factor in cleaning up the output, IMO.

For older code I've instructed it to delete all the comments, and then I re-comment it using a new session and these guidelines, asking it to rejustify the need for every comment to itself.

mmastrac··on DiffusionGemma Technical Report
That's mostly correct, but I believe the INT8 support on M5 changes the equation pretty drastically. I also believe that diffusion models generally output _better_ results if they have shaping for code (they often will miss delimiter matching inside of a canvas).
mmastrac··on DiffusionGemma Technical Report
You can save steps when diffusing from a pre-drafted canvas instead of seeding with random noise! If you have a drafter that is good enough, steps drop by ~50% or more, and tok/s can definitely increase.

I'd be curious to see your CUDA approach - happy to combine efforts as well.

mmastrac··on DiffusionGemma Technical Report
I re-implemented this one for macOS over the last couple of months: https://github.com/mmastrac/diffgemma

I like the model a lot and it's fairly good at reasoning. You can also really bend it to your needs. It's designed for machines with more compute than memory bandwidth but IMO does really well on metal.

I've got it up to ~15tok/s on M3-class machines, but I wager there's a bunch of perf on M5 that I just don't have hardware access to unlock.

I tried to implement MTP using the other Gemma MTP heads but I failed to move that perf needle. There's some interesting research to be done about pre-seeding the diffusion canvas from draft models. DiffusionGemma with the right drafter can hit 20-30 tok/s on my machine, but I've been unable to combine the two together to make it faster than what it's been running at so far.

mmastrac··on Show HN: Needle2: 14MB agentic LLM for phones, wearables, smart home and robots
Congrats on this release. The WASM implementation is really cool. This is a surprisingly good fit for a lot of cases, and I totally want to try turning this into a helper assistant for an application.

Please, though, take a pass at humanizing the text on the page. It's Clauded up all over and makes it hard to read.

mmastrac··on Inside vLLM: Anatomy of a High-Throughput LLM Inference System (2025)
I've been pondering a smaller quantization and experts swapping for this! Happy to take on PRs if you want to experiment as well.
mmastrac··on Inside vLLM: Anatomy of a High-Throughput LLM Inference System (2025)
I've been working on a fresh, AI assisted port of DiffusionGemma from scratch and it takes a significant amount of time to deslop. I've spend a nonzero amount of time on refactoring and comment-vomit cleanup.

https://github.com/mmastrac/diffgemma

mmastrac··on Does Speaking to Agents Like Cavemen Save 65% of Tokens? We Test
I suspect this means that there's just a ~10% inefficiency in token to information mapping.
mmastrac··on Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac
Which one? Nemotron Diffusion? It's impossible to say for sure, but I have a fairly deep library of metal kernels that _might_ cover some of the nvidia model's architecture.
mmastrac··on Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac
Awesome. I may need to finally bite the bullet and upgrade my macOS to test out the MPP approach you've taken.

I've got a number of tiled-load kernels, and a top-k attention kernel that you might find interesting.

mmastrac··on Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac
TBH, I think there's some truth to that. I spent _ages_ tuning the kernels to match the tested FLOP count of my M3's processor. I only have an M3 though and wasn't able to push int8 very far on it, but I think there's a chance that M5-class machines and higher might have more capability in this regard.

What I also learned is that MLX/vLLM is probably within ~20% or so of the absolute max perf on Mac. I found some improvements over what they were doing, but we're at the point where it's challenging to optimize without per-stepping kernels.

I found a few improvements over stock DiffusionGemma along the way, like using top-k attention, which drastically improves perf on my mac without sacrificing any of the benchmarks I was able to throw at it.

FWIW some of the issues with Gemma being slow on Mac are specific choices they've made in the architecture that make it challenging to make use various optimizations that have popped up recently. I think a Kimi K3-style network hybrid with the diffusion bits of DiffusionGemma could have some serious sway.

I think that diffusion still has an edge locally, but with some architecture tweaks and CPU improvements it would actually be a winner (ie: training the network for smaller token batch sizes or flexibility in attention heads, a less expensive attention mechanism, and others).

← PreviousPage 2 of 34Next →