ljosifov

Sprint outdial NUAs :)

Similar. Noticed DuckDB ever since an old article 'what db should I use' for local small data warehousing. The author was blown away that DuckDB seemed super naturally quick. It was I think columnar store + compression facilitated that? It made duckdb load compressed + on the fly decompress = faster than even reading the uncompressed data. Had forgotten most of it. But was used to mmap-ed files + columnar storing of Kdb. Was pleased that the author was clued to notice the power of that.

Then more recently I was given a somewhat random task to organise a motley collection of web scrapes, historic data, realtime data, data to be fetched on demand dispersed in semi-random collections. DuckDB as backing store + Claude Code that I discovered in Nov-2025 (with suitable skills and schemas) = a data agent where I could tell CC *in English* complicated data requests!? And CC would write glue shell and python code, write SQL and run it against DuckDB that housed most of the data, fetch new data, munge join filter, and present it to the user as "your data is in data slash blah". It seemed a miracle unfolded in front of my eyes! So yeah - fond of DuckDB. :-)

Latter I read this https://openai.com/index/inside-our-in-house-data-agent/ and thought "but of course".

Ah sorry - I misunderstood. Thanks for explaining it. Have not heard of Paseo nor Kepler, and have never tried Zed. Yeah I too assumed if I'm to try use OpenAI subscription outside Codex, or Anthropic subscription outside Claude Code - I'd get my account banned it's agains their rules. So I have never looked how using the whole harness from outside looks like either (except for 'claude -p'). Interesting. BTW I see now https://docs.z.ai/devpack/tool/codex Zai added OpenAI compatible end point.

I've used GLM-s the longest with Claude Code and their Anthropic supplied endpoint. As per their docs

$ ANTHROPIC_BASE_URL="https://api.z.ai/api/anthropic" ANTHROPIC_AUTH_TOKEN="zai-api-key" claude --dangerously-skip-permissions

Lately I use Zai in omp (oh-my-pi). It's listed built-in provider can be selected without configs shenanigans. Fits in the overall setup e.g. can select GLM-5.2 (now 5.3), and assign it role [plan] or [advisor]. I got reminded now of glm-5v-turbo. Think that 'v' was for vision. Assigned it role [vision] in omp now, let's see what happens. :-)

Wdym "sadly they don’t support using Claude Code"? For the longest time that's all Zai supported - Claude code. I'd run it via

  export ZAI_ANTHROPIC_BASE_URL="https://api.z.ai/api/anthropic"
  export ZAI_ANTHROPIC_AUTH_TOKEN="$ZAI_API_KEY"
  claude-zai() {
      { local -; set -x; } 2>/dev/null
      ANTHROPIC_BASE_URL="$ZAI_ANTHROPIC_BASE_URL" ANTHROPIC_AUTH_TOKEN="$ZAI_ANTHROPIC_AUTH_TOKEN" claude "$@"
  }
  $ claude-zai

I liked Claude Code to start with. But over time between 'CC cache thrashing undo' seetings (I see now accumulated in ~/.claude/settings.json) and Anthropic-anything becoming a liability - have not used it in while. ZCode is ok and use it to take advantage of the discount tokens on offer from time to time. But really glad to see that in omp (oh-my-pi) Zai is a 1st class provider, can be selected on it's own no configs shananigans needed. And fits in the overall picture. E.g. can select GLM-5.2 (now 5.3) assign role [plan] or glm-5-turbo [advisor].

Got reminded now of glm-5v-turbo - that 'v' was for vision - will try assign it role [vision] now in omp. See what happens. :-) Often times it's handy when describing gui problems if the harness/model 'can see'.

Atm OMP (oh-my-pi) with opencode-go subscription using deepseek-v4-flash-0731 (and mimo-v2.5 as advisor). Used pi before (still good, but too minimalist for all general use; works on Android under Termux btw) and OpenCode (which is ok too). I've used Codex the longest, but the $20 sub doesn't last that long. Before that Claude (and Claude on the web - about Nov'25), but nowadays with the new models not possible to do anything on the cheap sub. Got Zai (GLM-5.2) legacy sub to supplement too for variety. :-) Run local jobs on the 7900xtx (24gb vram - so dense models) batch bulk jobs. Non-dense models like nemotron-cascade-2-30b-a3b MoE run at 100 tok/s, looking forward to try the new lightning-3.5-30b-a3b. And got old MBP M2 Max 96gb uram (unified ram) for bigger sparse MoE models. Testing DS4F there under ds4 server/harness (too slow, 12-15 tok/s even with REAP25 reduced model), and (got agents) porting Ling-3.0-flash atm from llama.cpp -> ds4. Hoping only A5B (5B active, versus 13B of DS4F) will result in better tok/s. TBS. Back in Feb-Apr'26 looked local will be the only way to run llm jobs (plus 300w gpu job works out for office heating when cold), otherwise the ex-quota API payg costs would bankrupt me. But now with the oss weights models and providers like opencode - actually it's not only faster but also cheaper for me to run llm jobs on the api. Engines for local - llama.cpp for all, hipfire (mentioned) for the 7900xtx, and mlx on the apple ASI (that never quite work out for me). And ds4 trying forever when off-line :-) - e.g. on a flight - either ds4-agent, or ds4-server + agent pi. And I use Hermes as a general agent, non-coding agent, for everything that's not strictly coding.

Hipfire works on my 7900xtx best of all. Biggest surprise - Qwen3.6-27B dense does not grind to a halt with context depths all the way up to 250K! Measured at 1K, 8K, 32K, 64K, 128K, 192K, 250K - Hipfire speed holds close to 40 tok/s. Finished 5 days run of gpu 100% inferencing, Hipfire did not crash even once afaics. I assumed speed dropping like a stone on Qwen models was a feature/bug of the model, and "nothing can be done about it". Hipfire showed me wrong - pleasantly surprised there.

omp - current top, after using codex claude opencode pi that I still use too

Hear hear. IQ tokens to cheap to meter upon us. So many things changed since last week. Now I've had Prime agent session grinding into its 20-th hour still not giving up. Been using opencode-go since Go sub appeared. What made a difference was deepseek-v4-flash and mimo-v2.5 showing. Very similar middling models ~300b so light on the gpu. 1M context and hybrid archs - so one can actually make use of that 1M (don't grind to a halt like others). In OMP I have one the primary (default), the other one as /advisor looking over the shoulder and nagging. On opencode-go in credits counting they are the bottom-2 in cost, cheaper by 200-350 times than than the top-1. Last week with deepseek-v4-flash-0731 another jump - now it's closer to the top models then to the middle. Now I don't even need the /advisor probably. Still left it there it's sometime amusing the models back and forth. :-) DeepSeek offer /v1/responses api now with flash-0731, so setup Codex to use that too. I'm loving this :-)

Love pi. It's good to have a minimal agent, always welcome. Even if only to bootstrap install other agents. E.g. recently Hermes stopped installing under Termux (v0.19 - no, v0.18 - yes). Agent pi to the rescue: installed it under Termux (Android), started it, pointed pi to the Hermes github, asked for v0.18 specifically - and it did it. Cherry on the top - asked it to install a plugin from github I use (unmerged PR), did that too. Even if nowadays for coding I mostly use OMP (the pi with wheels and bells and whistles). For general computer use mostly use Hermes. Model-tied harnesses have their place e.g. Codex for me. I never know if it's allowed or banned using the sub from non-Codex harness - so stick with it. Even if only to have the 'canonical reference setup' to compare with and measure against.

Same. And I have come to use OMP (oh my pi) agent /advisor mode to put a 2nd model on the case (also mid-size one), reading everything. It can not block anything or change anything - just inserts comments in the text stream with 1 turn delay. Good portion of the time it's quiet. I'd say 1/2 of the time it's got something to say. About 2/3-rd of that the 'advice' is insubstantial or about something not-quite wrong. The good thing is the main model is confident - checks and then it stands its ground. Have not noticed it turning a right into a wrong b/c of advisor false alarm. And in 1/3-rd of the advice, it's a genuine defect teh advisor noticed, the main model works out a fix. This is my approximate feeling just observing the process, have not got collected the data. Afaik only OMP has advisor mode. Agent pi has plugin pi-omplike-advisor. For agent Hermes I had them code me an /advisor plugin (for now -0.1 old v0.18.x; yet to upgrade it to latest).

I usually doubt the 'small dataset tuned' variants. B/c ages ago (in the NN prehistory) I've done some NN training, and appreciate how hard it is to improve in general, and how easy it is to ruin a model in general while targeting a small dataset (LoRA-s are ok, that's different). That model/quant was the most recent one I was trying. But could not really use any of them, as the combo model + llama-server ground to a halt even at small context depth sizes on the amd gpu.

Yesterday I finally found a good combo! So writing this for the benefit for anyone that may have the same h/w. Got around to /GOAL search for something better for the h/w (amd 7900xtx), and pi agent found a new best that actually seems it will be useful for real. As the 40 tok/s speed starts dropping only at 260K context depth?? Served by hipfire from this repo https://github.com/Kaden-Schutt/hipfire, that worked the best got on llama-benchy:

  | Context | Wall Time | PP (t/s) | TG (t/s) |
  |:-------:|:---------:|:--------:|:--------:|
  |   1,024 |   17.3s.  |   660    |    40.8  |
  |   8,192 |   29.3s   |   606    |    40.8  |
  |  32,768 |   70.5s   |   599    |    40.7  |
  |  65,536 |  126.5s   |   594    |    40.6  |
  | 131,072 |  235.5s   |   591    |    39.0  |
  | 196,608 |  349.0s   |   591    |    40.2  |
  | 260,000 |  451.3s   |   594    |    37.2  |

This is - hipfire daemon dflash_mode auto, serve qwen3.6:27b --kv-mode asym2. Models:

  hipfire pull qwen3.6:27b           # 14 GB MQ4 quant, qwen3.6-27b.mq4
  hipfire pull qwen3.6:27b-draft     # 0.92 GB DFlash draft, qwen36-27b-dflash-mq4.hfq

Hobbled - but not to death, the few times I use it (usually on a plane). I tried 2bit of a 20% REAP reduced experts. :-O That's the biggest that fits on my own h/w (3yrs old M2 Max 96gb). It's coherent, it does work, doesn't fall apart on casual use. IDK if better than dense 27b. Think 27b was slower on the same h/w. DS4F has got 1M context window. Nowadays with weeks long run hermes sessions, I get to 300k-400k context depths easily. The speed decline profile of DS4F with context depth increase is superior to any other model I try. (I try them all - love this stuff) Only previous model coming close on that is nemotron-cascade-2 (only 30b-a3b) - that also has 1M context window.

True - they are workhorses. Not super bright, but good enough for lots of everyday tasks. I've found sweet spot to be turning thinking off, as it adds small or no value, while increasing the token count and waiting time. Last 27B I used was https://huggingface.co/Jackrong/Qwopus3.6-27B-Coder-GGUF - specifically post-train adapted a bit to run with thinking off. I saw today the 35B-A3B MoE from the same HF acc is out, downloading that rn to try.

Running 27B dense model on M5 128GB is ok, but one can do better.

On M5 128GB one can make use of the ram and use sparse MoE. For example, DeepSeek-V4-Flash will fit, served by DwarfStar (https://github.com/antirez/ds4). One will probably improve 2x the token/sec speed, given DS4F 13B activated params in the MoE are ~1/2 of the ~27B of the dense Qwen.

27B Of the Qwen fit even on a cheaper 24GB card, e.g. amd 7900xtx (<$1K?) or slightly dearer nvidia 3090 (with cuda). With ~900 GB/s bandwidth they will likely be ~50% faster than the M5 with 600 GB/s.

Haha :-) - FoxPro and Clipper next.

Not replaced but supplemented. For off-line coding current setup is pi + ds4-server + DeepSeek-V4-Flash REAP25 (on M2 Max 96gb). For simpler programming related (e.g. text2sql) as well as synthetic data generation, current best for me is llama.cpp + Gemma-4-26B-A4B (on gpu 7900xtx 24gb; sometimes nemotron-cascade-2-30b-a3b for 1M context). That and (dabbling now) auto-research uses lots of tokens. Used to get paused running out of token quotas all the time. The 1st local model I found somewhat useful to me was glm-4.7-flash, and it's gotten way better since. Recently between OpenCode Go choice of models at many price points, and DeepSeek-V4 dropping the IQ/$$$ by multiples, have become less reliant on local llms for this auxiliary work. Claude I use but with Zai GLM-5.2 subscription. And maintain GPT subscription for quality models.

For high Ram (unified), and relatively middling to lowish Tflops and bandwidth GB/s, usually MoEs are most hopeful. The current top-1 in the (iq, tok/s, @ context depth) ranks for me (M2 Max, 96gb) is DeepSeek-V4-Flash REAP25 <65gb gguf + ds4-server + pi agent. Not better than cloud API ofc, but useful enough to endure if I need to. E.g on a non-Internet 4h flight the battery (local llm draws 60w) held long enough. REAP supporting ds4 branch here

https://github.com/ljubomirj/ds4/tree/reap-compact-support

DS4F dropping to unusable <10 tok/s only at 784K context (!!) makes a big difference.

Yes, it's performant, and esp performant at non-trivial context depths. DeepSeek-V4 DS4 (and Flash - DS4F) drop tok/s speed much less than the rest. On my M2 Max it took context depths of 768K to drop tok/s to ~10 tok/s.

https://x.com/ljupc0/status/2062457314414587996

Other local models I've checked drop to unusable speeds way sooner. Only other model with similarity favourable curve I've tried is nemotron-cascade-2-30b-a3b. But it's a small model, way dumber than DS4F.

Coding agents use cases have large context depths. The rate of decline is as important as the headline number.

Thanks for the tear down. IDK anything about quantum (my knowledge there starts and ends with https://www.scottaaronson.com/democritus/lec9.html), but amused enough to follow in the background. See whether it ends crazy-bad or crazy-good. :-)

And just today now I saw this https://arxiv.org/abs/2606.07352. What's your take on it?

Am old enough to have witnessed more than one wave of "impossible things" happen for real in my lifetime, so lets see. As long as the scientific method (evidence, publication, replication, testing, etc) is mostly followed am curious enough to check blog posts or interviews from time to time.

+1 for boring. Boring code is Solid Code, in the sense of "Writing Solid Code" - the old book by Steve Maguire.

Thanks for the DS4, will give it a try. Was hoping maybe I can re-quantise shave few GB... MiniMax-M2.7 Unsloth's UD-IQ2_XXS is down to 65GB - it run albeit too slow to be usable to an agent at context depth. I'm curious DS4F with it being economical with the KV caches - if that translates into keeping up with context. Was hoping 80GB 2-bit quants maybe come down to 70GB... that would be more comfortable to run.

On 96gb I can give up to about 88GB to the GPU with sysctl iogpu.wired_limit_mb=88000, without suffering any ill-effects. When pushed higher I tend to notice e.g. graphic driver errors, youtube web page not working, other semi-random glitches. So the ~80 GB of DS4-flash quants I could just about fit. Leaving some extra for the KV caches. Will try, I'm curious how's the DS4 degradation with context depth growth, how fast does tok/s drop. E.g. 2-bit lowest quant MiniMax-M2.6 runs, but starts low tok/s and degrades fast with context depth.

The biggest models I can comfortably run are about 1/2 the DS4F size - like gpt-oss-120b. Lately was toying with Ling-2.6-flash. Got the agents to adapt existing metal kernels in llama.cpp, and it did run (model https://huggingface.co/ljupco/Ling-2.6-flash-GGUF, branch https://github.com/ljubomirj/llama.cpp/tree/LJ-Ling-2.6-flas...). It's 104B-A7B4, and for the M2 Max 7.4B active is about the most it can take while still producing 40 tok/s. And the hybrid arch allows for graceful degradation, still close to 30 tok/s at 64K context depth.

Too bad L2.6F while the best have, is not that much better in agentic benchmarks compared to my current incumbent local llm (nemotron-cascade-2). Got inspired by DS4 to start a l26f branch (WIP https://github.com/ljubomirj/l26f). :-) Try squeeze the most from L2.6F. There should be low hanging fruit in good integration of the agent and the inferencing engine. On input - considering the huge difference cached v.s. non-cached tokens. On output - considering that the NN gives us the complete logits set for all 200K+ tokens vocabulary.

Love this, even if can't use it atm (not got the h/w - only 96gb on M2 Max). I get it the general comp/public will find it unusable or worse. Reminds me of how home computers were - mere toys - before they became personal computers (PC). On my h/w the only passable combo for me atm is pi agent + llama.cpp + nemotron cascade-2 model: to 1M context, hybrid arch doesn't crash & burn 1/N^2 with context depths of 10K-50K-100K used by code agents. Was on a plane without Internet the other day. Brought a smile to my face that I could run pi agent (with llama.cpp serving), and it was just about usable at 40-30 tok/s. Afaik the usual API speeds are double that, 60-80 tok/s. Sensors showing using 60W when running inference. So battery probably would not last more than >3h. Model only 30B in size leaves plenty of space for KV-caches, and other programs - even at generous 8-bit quant. Only 3B active params at one time (with MoE A3B) is about the most that ageing M2 Max can carry it seems.

What we see and experience - it's all natural, it's the natural order. :-) When people claim something is un-natural, usually it's natural in that occurs in nature, only - they themselves find it objectionable. It's something they personally dislike, and would rather it not happen.

In the same boat with 7900xtx. 24GB vram, on paper decent performance, in reality most things don't run. Only llama.cpp is consistent that it can run most models, even if maybe not at top performance (afaik - lacking MTP, problems cache invalidation with hybrid models). At least with llama.cpp I know what runs. With various python-based inferencers, between their uv/venv, my venv, system envs/pythons/libs yadayada - I need an agent to get to the bottom of what's actually running. :-) Yeah IK skill issue/user errors - but don't have seconds in the day left to spend them on that.

Even if not perfect, if you publish on GH or HF, some other agent can maybe start there and not from zero. I did this for Ling-2.6-flash (107B-A7B4 MoE) that's the biggest llm I can ran for practical use on the other h/w I got for local llms (M2 Max). Even if MTP is not working well, still improvement on the current llama.cpp that does not run Ling-2.6-flash at all. This - https://huggingface.co/inclusionAI/Ling-2.6-flash/discussion.... The 4-bit quants are at https://huggingface.co/ljupco/Ling-2.6-flash-GGUF, the branch is at https://github.com/ljubomirj/llama.cpp/tree/LJ-Ling-2.6-flas....

~/llama.cpp$ build-.../bin/llama-batched-bench -m models/....gguf -npp 512,1024,2048,4096,8192,16384,32768 -ntg 128 -npl 1 -c 36000

  On amd 7900xtx

  Qwen3.6-27B-Q4_K_M
  |    PP |     TG |    B |   N_KV |   T_PP s | S_PP t/s |   T_TG s | S_TG t/s |      T s |    S t/s |
  |-------|--------|------|--------|----------|----------|----------|----------|----------|----------|
  |   512 |    128 |    1 |    640 |    0.743 |   689.35 |    4.605 |    27.80 |    5.348 |   119.68 |
  |  1024 |    128 |    1 |   1152 |    1.188 |   862.17 |    4.573 |    27.99 |    5.761 |   199.96 |
  |  2048 |    128 |    1 |   2176 |    2.566 |   798.09 |    4.602 |    27.81 |    7.168 |   303.57 |
  |  4096 |    128 |    1 |   4224 |    5.936 |   690.00 |    4.639 |    27.59 |   10.575 |   399.43 |
  |  8192 |    128 |    1 |   8320 |   15.034 |   544.90 |    4.729 |    27.06 |   19.763 |   420.98 |
  | 16384 |    128 |    1 |  16512 |   42.807 |   382.74 |    4.886 |    26.20 |   47.694 |   346.21 |
  | 32768 |    128 |    1 |  32896 |  137.377 |   238.53 |    5.188 |    24.67 |  142.566 |   230.74 |

  Qwen3.6-27B-IQ4_NL
  |    PP |     TG |    B |   N_KV |   T_PP s | S_PP t/s |   T_TG s | S_TG t/s |      T s |    S t/s |
  |-------|--------|------|--------|----------|----------|----------|----------|----------|----------|
  |   512 |    128 |    1 |    640 |    0.535 |   957.45 |    3.715 |    34.45 |    4.250 |   150.59 |
  |  1024 |    128 |    1 |   1152 |    1.124 |   911.16 |    3.677 |    34.81 |    4.801 |   239.97 |
  |  2048 |    128 |    1 |   2176 |    2.447 |   836.89 |    3.698 |    34.62 |    6.145 |   354.13 |
  |  4096 |    128 |    1 |   4224 |    5.711 |   717.17 |    3.729 |    34.32 |    9.441 |   447.43 |
  |  8192 |    128 |    1 |   8320 |   14.615 |   560.52 |    3.821 |    33.50 |   18.436 |   451.30 |
  | 16384 |    128 |    1 |  16512 |   41.966 |   390.41 |    3.967 |    32.26 |   45.933 |   359.48 |
  | 32768 |    128 |    1 |  32896 |  135.789 |   241.32 |    4.253 |    30.09 |  140.042 |   234.90 |

  On mbp M2 Max

  Qwen3.6-27B-UD-Q8_K_XL
  |    PP |     TG |    B |   N_KV |   T_PP s | S_PP t/s |   T_TG s | S_TG t/s |      T s |    S t/s |
  |-------|--------|------|--------|----------|----------|----------|----------|----------|----------|
  |   512 |    128 |    1 |    640 |    2.583 |   198.18 |   22.049 |     5.81 |   24.633 |    25.98 |
  |  1024 |    128 |    1 |   1152 |    8.321 |   123.06 |   22.364 |     5.72 |   30.685 |    37.54 |
  |  2048 |    128 |    1 |   2176 |   17.873 |   114.59 |   23.290 |     5.50 |   41.164 |    52.86 |
  |  4096 |    128 |    1 |   4224 |   41.967 |    97.60 |   23.624 |     5.42 |   65.591 |    64.40 |
  |  8192 |    128 |    1 |   8320 |   68.722 |   119.20 |   21.077 |     6.07 |   89.799 |    92.65 |
  | 16384 |    128 |    1 |  16512 |  142.184 |   115.23 |   22.026 |     5.81 |  164.210 |   100.55 |
  | 32768 |    128 |    1 |  32896 |  339.778 |    96.44 |   24.465 |     5.23 |  364.243 |    90.31 |

  Compared to similar prior models

  On amd 7900xtx

  Qwen3.6-35B-A3B-UD-Q4_K_S
  |    PP |     TG |    B |   N_KV |   T_PP s | S_PP t/s |   T_TG s | S_TG t/s |      T s |    S t/s |
  |-------|--------|------|--------|----------|----------|----------|----------|----------|----------|
  |   512 |    128 |    1 |    640 |    0.203 |  2517.60 |    1.482 |    86.35 |    1.686 |   379.67 |
  |  1024 |    128 |    1 |   1152 |    0.427 |  2399.22 |    1.471 |    87.04 |    1.897 |   607.15 |
  |  2048 |    128 |    1 |   2176 |    0.946 |  2165.23 |    1.478 |    86.59 |    2.424 |   897.67 |
  |  4096 |    128 |    1 |   4224 |    2.253 |  1818.33 |    1.502 |    85.22 |    3.755 |  1125.01 |
  |  8192 |    128 |    1 |   8320 |    5.849 |  1400.51 |    1.525 |    83.91 |    7.375 |  1128.17 |
  | 16384 |    128 |    1 |  16512 |   17.115 |   957.27 |    1.589 |    80.55 |   18.705 |   882.78 |
  | 32768 |    128 |    1 |  32896 |   56.008 |   585.06 |    1.704 |    75.10 |   57.712 |   570.00 |

  Qwen3.6-35B-A3B-UD-IQ4_XS
  |    PP |     TG |    B |   N_KV |   T_PP s | S_PP t/s |   T_TG s | S_TG t/s |      T s |    S t/s |
  |-------|--------|------|--------|----------|----------|----------|----------|----------|----------|
  |   512 |    128 |    1 |    640 |    0.204 |  2508.94 |    1.313 |    97.46 |    1.517 |   421.78 |
  |  1024 |    128 |    1 |   1152 |    0.423 |  2418.64 |    1.296 |    98.80 |    1.719 |   670.18 |
  |  2048 |    128 |    1 |   2176 |    0.946 |  2164.61 |    1.323 |    96.78 |    2.269 |   959.13 |
  |  4096 |    128 |    1 |   4224 |    2.235 |  1832.54 |    1.326 |    96.52 |    3.561 |  1186.06 |
  |  8192 |    128 |    1 |   8320 |    5.845 |  1401.44 |    1.352 |    94.70 |    7.197 |  1156.03 |
  | 16384 |    128 |    1 |  16512 |   17.096 |   958.38 |    1.417 |    90.33 |   18.513 |   891.94 |
  | 32768 |    128 |    1 |  32896 |   56.013 |   585.00 |    1.530 |    83.66 |   57.543 |   571.67 |

  Carnice-Qwen3.6-MoE-35B-A3B-Q4_K_S
  |    PP |     TG |    B |   N_KV |   T_PP s | S_PP t/s |   T_TG s | S_TG t/s |      T s |    S t/s |
  |-------|--------|------|--------|----------|----------|----------|----------|----------|----------|
  |   512 |    128 |    1 |    640 |    0.205 |  2499.78 |    1.483 |    86.31 |    1.688 |   379.16 |
  |  1024 |    128 |    1 |   1152 |    0.434 |  2361.36 |    1.448 |    88.40 |    1.882 |   612.25 |
  |  2048 |    128 |    1 |   2176 |    0.947 |  2161.87 |    1.478 |    86.62 |    2.425 |   897.27 |
  |  4096 |    128 |    1 |   4224 |    2.259 |  1813.00 |    1.472 |    86.94 |    3.732 |  1131.98 |
  |  8192 |    128 |    1 |   8320 |    5.892 |  1390.42 |    1.505 |    85.06 |    7.397 |  1124.85 |
  | 16384 |    128 |    1 |  16512 |   17.397 |   941.77 |    1.568 |    81.61 |   18.965 |   870.63 |
  | 32768 |    128 |    1 |  32896 |   56.296 |   582.07 |    1.690 |    75.74 |   57.986 |   567.31 |

  Nemotron-Cascade-2-30B-A3B-IQ4_XS
  |    PP |     TG |    B |   N_KV |   T_PP s | S_PP t/s |   T_TG s | S_TG t/s |      T s |    S t/s |
  |-------|--------|------|--------|----------|----------|----------|----------|----------|----------|
  |   512 |    128 |    1 |    640 |    0.195 |  2622.33 |    0.972 |   131.69 |    1.167 |   548.30 |
  |  1024 |    128 |    1 |   1152 |    0.407 |  2514.76 |    0.934 |   137.10 |    1.341 |   859.16 |
  |  2048 |    128 |    1 |   2176 |    0.854 |  2396.99 |    0.942 |   135.90 |    1.796 |  1211.42 |
  |  4096 |    128 |    1 |   4224 |    1.895 |  2161.89 |    0.953 |   134.36 |    2.847 |  1483.50 |
  |  8192 |    128 |    1 |   8320 |    4.593 |  1783.70 |    0.967 |   132.43 |    5.559 |  1496.60 |
  | 16384 |    128 |    1 |  16512 |   12.213 |  1341.53 |    0.996 |   128.56 |   13.209 |  1250.10 |
  | 32768 |    128 |    1 |  32896 |   36.998 |   885.66 |    1.059 |   120.89 |   38.057 |   864.39 |

  On mbp M2 Max

  Qwen3.6-35B-A3B-UD-Q6_K_XL
  |    PP |     TG |    B |   N_KV |   T_PP s | S_PP t/s |   T_TG s | S_TG t/s |      T s |    S t/s |
  |-------|--------|------|--------|----------|----------|----------|----------|----------|----------|
  |   512 |    128 |    1 |    640 |    0.540 |   947.31 |    2.489 |    51.42 |    3.030 |   211.22 |
  |  1024 |    128 |    1 |   1152 |    0.951 |  1077.21 |    3.237 |    39.54 |    4.188 |   275.10 |
  |  2048 |    128 |    1 |   2176 |    2.994 |   684.10 |    3.139 |    40.77 |    6.133 |   354.80 |
  |  4096 |    128 |    1 |   4224 |    6.245 |   655.85 |    3.210 |    39.88 |    9.455 |   446.75 |
  |  8192 |    128 |    1 |   8320 |   12.411 |   660.08 |    3.284 |    38.98 |   15.694 |   530.13 |
  | 16384 |    128 |    1 |  16512 |   28.321 |   578.51 |    3.584 |    35.71 |   31.905 |   517.53 |
  | 32768 |    128 |    1 |  32896 |   65.725 |   498.56 |    4.029 |    31.77 |   69.754 |   471.60 |

  Nemotron-Cascade-2-30B-A3B-Q8_0
  |    PP |     TG |    B |   N_KV |   T_PP s | S_PP t/s |   T_TG s | S_TG t/s |      T s |    S t/s |
  |-------|--------|------|--------|----------|----------|----------|----------|----------|----------|
  |   512 |    128 |    1 |    640 |    0.528 |   969.13 |    2.036 |    62.87 |    2.564 |   249.59 |
  |  1024 |    128 |    1 |   1152 |    1.079 |   948.84 |    3.201 |    39.99 |    4.280 |   269.15 |
  |  2048 |    128 |    1 |   2176 |    3.390 |   604.10 |    2.952 |    43.36 |    6.342 |   343.11 |
  |  4096 |    128 |    1 |   4224 |    6.756 |   606.28 |    2.991 |    42.79 |    9.747 |   433.35 |
  |  8192 |    128 |    1 |   8320 |   13.647 |   600.30 |    3.061 |    41.81 |   16.708 |   497.97 |
  | 16384 |    128 |    1 |  16512 |   29.491 |   555.56 |    3.414 |    37.50 |   32.905 |   501.81 |
  | 32768 |    128 |    1 |  32896 |   65.867 |   497.49 |    3.663 |    34.95 |   69.530 |   473.12 |

Dang I saw some lowish numbers there for Spaks (and Strix). As I was eyeing a spark to get some CUDA exposure... :-O

Glad to see other people using it. Saved my life, was going crazy click-clicking to nab the right window. Now Cmd-1..9 brings to focus a window of my chosen application. (Chrome) In case it helps someone else, myself and Codex iterating over time https://github.com/ljubomirj/dotfiles/blob/main/.hammerspoon.... Cmd-1..9 switches over focuses to a particular window, Cmd-0 presents an (ugly; but suffices) dialog box to select the window with arrows (of the App of interest - Chrome for me atm) to switch to. But more important - to see what window what Window name is recalled by the particular Cmd-1..9 shortcut. Option-arrows shuffle window-to-key ordering. I right-click-Name Window my windows. Think back now - on restart they may even be preserved?? Don't recall re/naming them manually recently. (possible I've forgotten though)

Say more please if you can. How/why is ik_llama.cpp faster then mainline, for the 27B dense? I'd like to be able to run 27B dense faster on a 24GB vram gpu, and also on an M2 max.

Everyone should do the calculation for themselves. I too pay for couple of subs. But I'm noticing having an agent work for me 24/7 changes the calculation somewhat. Often not taken into account: the price of input tokens. To produce 1K of code for me, the agent may need to churn through 1M of tokens of codebase. IDK if that will be cached by the API provider or not, but that makes x5-7 times price difference. OK discussion today about that and more https://x.com/alexocheema/status/2020626466522685499