HNHacker News
TopNewBestAskShowJobs

gcr

6,920 karma · joined January 1, 2010

submissionscomments
gcr··on Qwen3.7-Max: The Agent Frontier
on my M1 Max, MTP consistently lowers my performance! I’ve tried both llama-cpp’s recently landed MTP support (cloned and built Tuesday) as well as one of the other forks a few weeks ago. Suspect nobody’s done a comparison on hardware like mine.
gcr··on Qwen3.7-Max: The Agent Frontier
Pardon. You had mentioned open weight models so I assumed you meant self-hosted
gcr··on How fast is N tokens per second really?
Agreed. Prefill kills me for local model work. The model reads much faster than it writes, but I'd love to get a sense for how fast the model can read large source conversations.
gcr··on Qwen3.7-Max: The Agent Frontier
Depends. ROCm is pretty well-supported for example.

Non-NVIDIA backends tend to get less support and new features land slower, or features that are expected to improve performance wind up hurting it instead. That sort of thing.

For basic “token in/token out” workloads without fine tuning, it’s probably fine ??

gcr··on Google changes its search box
Why is this answer notable to you?

Wouldn’t you be more surprised / concerned if the model refused to to criticize Google?

gcr··on Qwen3.7-Max: The Agent Frontier
how could running the qwen GGUF phone home? that would require cooperation with the inference backend (llama-cpp), or some kind of model exploit. It’d be far easier to pay the agent harness devs or supply-chain some plugin or something, that space is the Wild West anyways

I've certainly used these models without wifi without any differences.

gcr··on Qwen3.7-Max: The Agent Frontier
Look for an open release from the Qwen team in the coming weeks. They like to showcase their proprietary models first, which score higher on benchmarks anyway due to model size.
gcr··on Qwen3.7-Max: The Agent Frontier
There are two flavors of Qwen 3.6:

- A 27B "dense" model

- A 35B "Mixture of Experts" model, which activates only 3B parameters for each token.

For your hardware, I strongly recommend `unsloth/Qwen3.6-35B-A3B-GGUF:Q4_K_M`. I have an M1 Max with 32GB VRAM from 2021 that can read at ~300-500 tokens/sec and write at ~30 tokens/sec with llama-cpp's default settings, which is plenty fast. The 27B model can read ~70tok/sec and write ~5tok/sec.

The 35B MoE model technically takes slightly more memory but is much faster because it's doing 1/9th the work. It's not quite as "smart", but it's comparable.

gcr··on Qwen3.7-Max: The Agent Frontier
here's a simple setup to get you started on an Apple M1 Max from 2021 with 32GB VRAM. it will download 20GB of models to `~/.cache/huggingface/hub`, which you can delete when you're done.

  /Users/gcr/llama.cpp/build/bin/llama-server
      -hf unsloth/Qwen3.6-35B-A3B-GGUF:Q4_K_M
      --no-mmproj-offload
      --fit on
      -c 65536 # edit to taste
      --reasoning on --chat-template-kwargs '{"preserve_thinking": true}'
      --sleep-idle-seconds 90 # very aggressive: purge model from vram after this long
      -ctk q8_0 -ctv q8_0 # Optional. Lower memory use, but lower speed. Omit if you can.
I don't recommend ollama or lm-studio. Ollama's in the process of switching from their llama-cpp backend anyway, but their new go framework frequently OOMs and crashes on my hardware. I also don't recommend MLX-based inference backends on this hardware; I've found them to consistently reduce performance, contrary to what I've read online. I've tried all the llama-cpp metal forks, but right now, MTP, TurboQuant, MLX, etc etc etc are too new and just slow things down. It's all dust in the wind still.

For agent harnesses, opencode is okay, as is pi or even Zed's built in agent panel. Claude code "works" with ANTHROPIC_BASE_URL=http://localhost:8080/v1, but is very chatty (the default system prompt burns 20k tokens). Crush (from the charm-bracelet folks) is particularly nice when starting out. I've personally converged on pi-agent under an otherwise-mostly-default setup. You can ask qwen to customize pi or write you an extension which helps a little.

You'll need to add `http://localhost:8080/v1` as an OpenAI-compatible model provider in your coding harness with any API key (doesn't matter) and any model identifier (doesn't matter with llama-cpp).

Note that pi doesn't have permissions. Everything is permitted. The hundred hungry ghosts you've trapped in a jar WILL find a way to delete your home folder someday. That's what Man gets for summoning demons without casting a circle of protection first. Flying too close to the sun etc etc etc

Take backups and then go have fun. Hope this helps.

gcr··on Qwen3.7-Max: The Agent Frontier
What’s the price point for getting into that sweet spot?

I’m on an M1 Max with 32GB VRAM, so I’m looking forward to the 27B or 35B-A3B models. Is dropping $5k for an RTX 6000 or a DGX Spark really the best option?

gcr··on Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
how do you know your harness design isn’t just overfitting on your test set?
gcr··on Incident Report: Railway Blocked by Google Cloud [resolved]
> Railway owns our vendor choices, and we ultimately own this one. Your customers don't care whether the failure was Google or Railway; they see your product. Your uptime is our responsibility, and we'll keep delivering on it.

This is an excellent closing statement.

gcr··on The last six months in LLMs in five minutes
They’re talking past each other. For some, “high quality” is a comment about implementation elegance. For others, “high quality” is about duct-taping crude implementations together to fashion a kickass user experience. To most, quality probably involves some convex combination of these.
gcr··on Eric Schmidt speech about AI booed during graduation
Exactly! Thats a complimentary problem for new grads
gcr··on Eric Schmidt speech about AI booed during graduation
Your last sentence is factually not true. Friends across five different companies report that LLM adoption is now a key metric in their performance review.
gcr··on Eric Schmidt speech about AI booed during graduation
For college graduates, LLM tech is an existential threat to their livelihoods by making it so much harder to start a career without connections or pedigree.
gcr··on Show HN: Find the best local LLM for your hardware, ranked by benchmarks
canirun doesn’t work well for MoE models. Qwen’s 35B A3B model works amazing on my M1 Max 32GB, but canirun thinks it barely runs.
gcr··on Bitcoin trader recovers wallet with help of Claude
There’s some weak evidence against this actually. Harness design makes a huge difference for tiny local models for example: https://itayinbarr.substack.com/p/honey-i-shrunk-the-coding-...

I have only my own personal experience for frontier models, but I have seen different performance of Opus when used from Pi or Claude Code or Zed for example.

gcr··on A few words on DS4
DwarfStar4 is a small LLM inference runtime that can run DeepSeek 4. The blog post implies that it currently requires 96GB of VRAM.

For others who are lacking context :-)

gcr··on Bitcoin trader recovers wallet with help of Claude
When I hear "claude code one-shotted X" and X is a novel problem, I mentally substituted "the agentic harness that I tried one-shotted X," since that's what they're saying.

Getting any smart model to take a look at the task is the sort of lift that the speaker is usually pointing to.

gcr··on Bitcoin trader recovers wallet with help of Claude
That's what the comment is saying. In normal repositories, version control acts as a record of the momentum of the direction the product was taking. If it's just "_old" and "_new," the developer has to read and understand both, which I think is going to be far more time consuming than your estimation.
gcr··on Twin brothers wipe 96 government databases minutes after being fired
Google powerwashes your corp Chromebook when they let you go. A friend was composing an email on the train when their screen went black and the device reset itself to factory settings.

They even send the “you’re being fired” email to their personal email they have on file. Didn’t even schedule a meeting.

gcr··on Claude subscription changes coverage of `claude -p`
You're not understanding the difference between API billing and subscirption billing.

Subscription usage refills every five hours. API billing doesn't.

gcr··on Claude subscription changes coverage of `claude -p`
Ya but that monthly usage doesn't refill every five hours or every week the way your subscription usage does. Once you're through the $200, you're done until next month.
gcr··on Claude subscription changes coverage of `claude -p`
As a fellow zed user, I fear this could affect us (though i'm not 100% confident). Zed uses their own adapter to implement claude's ACP support, and it's likely using `claude -p` (the agent SDK) under the hood.

Hoping the Zed team will clarify.

gcr··on Claude subscription changes coverage of `claude -p`
A better link: https://support.claude.com/en/articles/15036540-use-the-clau...
gcr··on Restore full BambuNetwork support for Bambu Lab printers
This is HP’s current philosophy towards consumer desktop inkjet and laser printing, and customers universally hate it. No thanks!
gcr··on Ratty – A terminal emulator with inline 3D graphics
How does that work with all the kitty terminal protocol image stuff if I have plots to display?
gcr··on CERT is releasing six CVEs for serious security vulnerabilities in dnsmasq
MaraDNS is much less popular than dnsmasq though.

I have several libraries that I've written. Not one single serious security bug in them has been found since 1991. Granted, nobody uses my libraries...

Not to diminish your team's achievement! :D But it's important to contextualize claims like this with information about what your userbase looks like

gcr··on I returned to AWS and was reminded why I left
In the USA, small companies don’t generally have the resources to take large companies to court, even if they’re in the right.
← PreviousPage 5 of 34Next →