Comment by OtomotO | Hacker News Reader

Comment by OtomotO | Hacker News Reader

Parent Full thread

I need to play around with this.

I am running on a Ryzen 5950X, 64GB DDR4 Ram and a Radeon RX 6700 XT with 12GB... I don't expect wonders, but I have between 11 and 120 (!) t/s.

I told it an open ended question: "What would you improve in this file at {location} in {bigger project}" and the context window seems to be too small. It's running in circles.

I must be doing something wrong.

Again, I am not expecting wonders and I am totally willing to accept that "I am holding it wrong".

llama-server \ -m ~/models/tiel-coder/Tiel-Coder-35B-A3B-MTP-UD-IQ4_XS.gguf \ -ngl 20 \ -c 16384 \ --jinja \ --alias tiel-coder

using it in opencode

{ "$schema": "https://opencode.ai/config.json", "provider": { "llamacpp": { "npm": "@ai-sdk/openai-compatible", "name": "Local llama.cpp", "options": { "baseURL": "http://127.0.0.1:8080/v1" }, "models": { "tiel-coder": { "name": "Tiel-Coder 35B A3B", "limit": { "context": 16384, "output": 4096 } } } } }, "model": "llamacpp/tiel-coder" }

Ideally you want to find something that allows a context window for at least 65k for agent work. Over 100k is better but 32k is about the bare minimum for anything in a harness/agent like Pi. You might have to drop to a lower quantisation to make that possible but context is the key for longer lots of thinking questions like this.

If you drop the thinking level to low or even off the context will go further at the cost of the models intelligence. That might also help you get some answer but since such a question will read so many files into that context its the wrong approach. This is one of the bigger limitations of local LLMs, you can't do work that spans across a lot of files because you are almost always context limited. There is a limit of 1M even with the frontier so even that can't review an entire project. Focussing in on individual files or small groups is more achievable.

So just to be specific the -c needs to be >65536 and depending on the size of what is in the folder might very well not be enough.

Your setup is definitely a monster compared to mine so it should give much better results. I also believe you run dense models with decent results with that setup.

Not sure what llamacpp build you are using but on mine using this fork: https://github.com/TheTom/llama-cpp-turboquant on commit 407f3237bfb3eeaff61546797de3d8c1a96be748 compiled for Pascal, I am able to push my current system to 131k context with minimal loss in quality. For reference this is my config.ini file for this model (Cyber is the better uncensored version btw):

[*] # Global defaults jinja = 1 metrics = 1 threads = 12 threads-batch = 12 flash-attn = on sleep-idle-seconds = 99999 timeout = 99999 no-mmap = 1 mlock = 1 ctk = turbo4 ctv = turbo2

[qwen36-a3b] model = /root/llm/models/Cyber-Tiel-Coder-35B-A3B-MTP-UD-IQ4_XS.gguf alias = qwen36-a3b n-cpu-moe = 32 temp = 0.6 top-p = 0.95 top-k = 20 min-p = 0.0 presence-penalty = 0.0 repeat-penalty = 1.0 reasoning = on n-predict = 32768 reasoning-format = deepseek reasoning-budget = 16384 reasoning-budget-message = "Proceed to final answer" reasoning-preserve = true swa-full = true np = 2 batch-size = 4096 ubatch-size = 2048 cache-reuse = 256 ngl = auto ctx-checkpoints = 33 ctx-size = 262144 no-mmproj = true no-mmproj-offload = true spec-type = draft-mtp spec-draft-n-max = 2

I am also certain that these flags could be improved somehow depending on your system.

I would give it significantly more context, if you don't have enough room you can quantize the kv cache with some flags that I don't remember ottomh.

I would go ahead and give it at least 64k but the more the better really!

I run the kv cache at q8 for keys and q4 for values, but going below q8 is generally not recommended.

--cache-type-k q8_0 --cache-type-v q8_0

would drop it from fp16 to 8 bits. You don't want to go below 8 bits if you can possibly avoid it and if you really do need to squeeze just drop the v value to q4_0 but leave the key alone. Unlike model quantisation this really makes things worse quickly.

Yeah, you need both a smaller system prompt than OpenCode uses, and a larger context configured in llama.cpp. It's probably running into compaction cycles before it even gets to start thinking about the task at hand.

You'd have better luck with pi instead of OpenCode, and see how far you can get that context window higher. For any halfway actual work you need bare minimum 64k, ideally 128k+. You can use 8bit kv cache, despite whatever folklore you may hear otherwise.

Reply on news.ycombinator.com