apitman

It's Netflix before the unbundling happened

That's pretty sweet. Thanks

I'm sometimes tempted to use these sorts of proxies, but I always worry that the hacks they have to use to wrap the upstream tools and APIs is going to mess with my downstream harness/agent.

What do they do with old ones? Their hardware physically can't run other models right?

You can turn on various levels of some of many of the models in the UI

Seems like that equation really should weight total params and active params differently

Very interesting. I was not expecting anything close to this.

Currently software is king when it comes to inference performance though.

You should be getting way more than that on a 6000 pro even today. I'm getting 40tok/s on a pair of 3060s. You can ask a SOTA model to optimize your setup for you.

Aren't things like KV size inherent to the model?

I wonder how this would stack up against 4x RTX 3060, assuming you have the physical room for them.

Interesting. Why don't the unsloth guides (https://unsloth.ai/docs/models/qwen3.8) mention this? Do they already include the fixes in their GGUFs?

Check out OpenCode Go as well. They give some Kimi K3, Qwen3.8 Max, and GLM5.2 (probably 5.3 soon?) usage which may cover your needs for $10/mo

Yeah make sure you're using MTP and potentially tensor parallelism.

This is specifically about unsloth having day-1 GGUF's available.

I used GPT-5.6 Sol high to optimize it, and it claimed it was getting 50. I'm seeing ~40 on my goto smoketest: "Make me a vector add in CUDA".

Funny side note. It successfully one shot the program, but it wasn't able to run it because there literally wasn't enough VRAM left to allocate CUDA memory. Watching it try to debug that was fascinating. I'm pretty sure it would have killed the llama-server (and thus itself) if it hadn't been running in a separate container.

Running it on 2x3060 now. Works pretty well but VRAM is tight. 4bit quants. 1x128k context, 8bit KV, MTP on.

The Luna price dropped the day before a massive update to Flash (0731 update). GP may be referring to that version.

Look at OpenRouter. They have lots of providers

But why? Luna Max is almost the same intelligence as Terra xhigh and way way cheaper. And Terra max is almost the same as Sol high. I just don't really see a place for Terra but slower.

Wait people use terra?

Are there any projects that track how much usage of each model translates to how much percentage drop in weekly/5hr windows?

This is what I do. Nice benefit is it lets me passthrough my GPU and share it between multiple containers. Incus is awesome.

Fun fact: QEMU runs natively on windows and supports acceleration with WHP. It works surprisingly well.

> OpenCode Go currently offers $120 for $10 on DeepSeek Flash v4

At DeepSeek's absurdly low rates or market rates?

I'm getting like 25 tok/s on 2x RTX Pro 6000. This is with llama.cpp, but I had GPT tune it for me. I was under the impression vLLM was at most ~2x faster, and usually for highly parallel loads. Any tips on where I should look first for an obvious blunder?

I'm guessing tensor parallelism or similar?

These are very interesting results, and honestly hard to believe, even as a big 0731 fan.

If I'm reading the chart correctly, a couple observations:

* deepseek-v4-flash-0731 max is better than kimi-k3 max

* glm-5.2 is dumber than a box of rocks (this must be on low reasoning or something, right?)

This is way more extreme than other results I'm seeing, like those from Artificial Analysis.

I've found it to be pretty good so far.

DeepSeek has far cheaper cache pricing. That's the difference.