Hacker News Reader

I have a old supermicro X10DRU-i server with two Tesla V100's (48 GB VRAM) and 128GB RAM and have been running qwen3.6-27B with a lot of success. I would say it's performance on my use case (modifying and extending a ~70kloc C++ code base) has been excellent. I have no benchmarks, but it seems comparable to claude sonnet 4.6 in capabilities. I run it with llama.cpp:

llama-server -m Qwen3.6-27B-Q8_0.gguf -c 131072 --tensor-split 0.4,0.6 --batch-size 256 --cont-batching --flash-attn on -ngl 999 --threads 16 --jinja

I regularly get ~22tok/s when context utilization is below <65k, but it does slow done to ~13tok/s when the context is nearly full (lots of swapping to RAM). I have been using the qwen-code harness though, since it is far more token efficient than claude-code which injects massive prompts that chew up the context window. I plan on trying it with pi next.

I'm keeping my ~$20/mo claude subscripts for the planning prompts, and then hand it off to qwen for implementation. It's been working well so far.

This link [1] features some good insight on how to adapt your usage to smaller models which require more explicit or deliberate prompting. I have been using Gemma 4 31B a lot and have found it very competent. It can be a bit unstable and start spiraling or end up in infinite loops that you need to reset, but for the most part it's been really good.

[1]: https://itayinbarr.substack.com/p/honey-i-shrunk-the-coding-...

thanks for the link! Interesting paper and little-coder looks purpose built for my own local model agentic experiments.

I can see that and I don't know your setup, but there are people pushing >70t/s with MTP on a single 3090, with big contexts still >50t/s. 64k is not a lot for agentic coding, and IIRC 128k with turboquant and the likes should be possible for you. r/LocalLLM/ and r/LocalLLaMA/ are worth a visit IMO.

EDIT: just found this recipe repo, may wanna give it a go: https://github.com/noonghunna/club-3090

EDIT-2: this can also shave off a lot of context need for tool calling -> https://github.com/rtk-ai/rtk

will give more info in the post

EDIT: thanks for the links!

I managed to execute with vllm successfully, but it breaks opencode on simple "what's this repo?" task. On oh-my-pi it wont event execute because omp sends multiple system prompts. I'll try with llama.cpp later and see if it works more reliably.

club-3090 with llamacpp did it. Full 262k context, usable in oh-my-pi. Still testing it, but initial results are promising.

I had to make a couple of adjustments though. After downloading the model with hf, I needed to move the mmproj-F16.gguf to the parent folder:

   tree /media/fast-storage/club-3090-models/qwen3.6-27b/
  /media/fast-storage/club-3090-models/qwen3.6-27b/
  ├── mmproj-F16.gguf
  └── unsloth-q3kxl
      └── Qwen3.6-27B-UD-Q3_K_XL.gguf

then, on starting the server, the container would complain that llama-server wasn't a known binary, so I needed to add PATH="/app:$PATH" to the entrypoint of the llama service.

The only things that's missing is for llama to emit thinking blocks that oh-my-pi can parse, but it's running alright. That's mostly cosmetic.

That‘s so cool man! Congrats!

For context, I'm feeling like I have a "free Sonnet" now that I've got Qwen3.6 35B running on my 5070ti at home (I connect to it via Tailscale). I run it _almost exactly_ the same as this Reddit post which found a good way to squeeze the 35B model onto a GPU with 16GB of VRAM: https://www.reddit.com/r/LocalLLaMA/comments/1sor55y/rtx_507... I really like it because it's slightly more operationally complex (I had to write a script to start it) but now that I have it, I literally never have to change it. It's a folder with the llama-server in it and with the model.gguf in it, I run the script which starts serving the model, done.

Like that post, I get 75 tokens/second. The exact model is: Qwen3.6-35B-A3B-UD-Q4_K_M.gguf and I get 128k of context

I run it on my home machine and connect to it from anywhere over tailscale. I connect through the opencode CLI which I configure with this as provider by adding the following to my `~/.config/opencode/opencode.json`:

    {
      "provider": {
        "vllm": {
          "npm": "@ai-sdk/openai-compatible",
          "name": "local-llm-qwen3.6-35B",
          "options": {
            "baseURL": "http://homepc.tail987654.ts.net:8033/v1"
          },
          "models": {
            "Qwen3.6-35B-A3B-UD-Q4_K_M.gguf": {
              "name": "Qwen3.6-35B"
            }
          }
        }
      }
    }

You're not sharing what quantization you're using, in my experience, anything below Q8 and less than ~30B tends to basically be useless locally, at least for what you typically use codex et al for, I'm sure it works for very simple prompts.

But as soon as you go below Q8, the models get stuck in repeating loops, get the tool calling syntax wrong or just starts outputting gibberish after a short while.

will do that in an edit to the post

Sure, waiting :)

In the meantime, Ollama seems to default to "Q4_K_M" which is barely usable for anything, and really won't be useful for agentic coding, the quantization level is just too low. Not sure why Ollama defaults to basically unusable quantizations, but that train left a long time ago, they're more interesting in people thinking they can run stuff, rather than flagging things up front, and been since day 1.

I'm currently giving club3090 a try, it seems to have lots of pre-configured setups depending on the workflow. I'm trying vllm first, then with llama.cpp.

Ollama is definitely not the way to go once you have an interest beyond "how quickly can I run a new LLM" rather then "how do I use a local llm to do things in a remotely optimal way"

Yeah. Context size matters a lot. With OpenCode dumping like 10k tokens in the system prompt it takes like 4 rounds before it had to compact at say 64k. It's not really worth it to run at anything below 100k and even then the models aren't all that useful.

They're also pretty terrible at summarization. Pretty much always some file read or write in the middle of the task would cross the context margin and it would mark it as completed in the summary. I think leaving the first prompt as well as the last few turns intact would improve this issue quite a lot, but at low context sizes thats pretty much the whole context ...

I see your updated post. Switch over to llamacpp and look up recommended quants and settings. A good place for this info is on /r/localllama

Yep! I'm currently trying vllm, then I'll give llamacpp a try too

When you run ollama serve, make sure you override the context size to about 32K. Also, I give the model a useful short README.md on the code it is writing or modifying, and a Makefile with useful targets for the agent to use. I usually use Claude Code with qwen3.6

I also go outside for fresh air while I wait for a session to run.

Something promising I found is "DFlash DDtree Qwen3.5 & Qwen3.6 27B GGUF on RTX 3090" - https://github.com/Luce-Org/lucebox-hub

Didn't know 3.6 was available on Ollama outside of MacOS!

The (unsloth dynamic) 4-bit quants of Qwen 3.6 kept getting stuck in circles for me. Even though it doesn't benchmark as well, GLM 4.7 Flash at least keeps making progress if I keep nudging it, so I have actually been able to finish some apps with it.

Qwen3.6 supports 266k context out of the box. Try using q8 kv cache to enable more of it.

I limited it to 64k expecting 24GB vram to not be enough to make use of the entire context window, but I'll try with other's suggestions.

I agree for planning it's not there yet. But I wouldn't be surprised if something came out that was in a similar weight class.

> qwen

I only have luck with pi and qwen bashing 100 line scripts. Everything real needs a planful model. To your point:

> You need to pair it today with a model capable of ingesting the whole context and providing a detailed plan…

Curiously, ANTHROP\C seems determined to ensure you don't use your Opus 4.7 Max 1M tokens for this any more, instead it sics Haiku on your context to "sample" using a weird pile of inchoate regex tp return "no more than 50 lines" or similar uselessness then finally Opus goes and burns tokens cogitating a solve for a problem shape that doesn't have anything to do with the areas of interest, inevitably unsampled.

I really really really want a "no subagents, no sampling" mode (telling it all subagents are Opus in env vars doesn't seem to persuade it to go ahead and use those 1M tokens to just, you know, read the damn file. Ironic if getting the best out of Opus cannot be in their harness.

All this said, it seems most people think AI saves them time and money so long as it costs no money — feels like ANTHROP\C is optimizing for that. I get it.

But can we also have a `ANTHROPIC_ENABLE_HIGH_ROI=1` mode please?

It costs more fixing all these unnecessary oversights than it would cost to just do the toil the machine is here to do.

Try oh-my-openagent plan mode.