Hacker News Reader

gaeldon Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

True, and for third-party models we'll just re-use their public open weights.

There is a time-consuming part, though, that is performed manually by our (human) team: implement the logic of the model in C++ and assembly code in a super-optimized way, co-designed for each specific hardware card.

This can take months.

We hope to accelerate the process with AI agents, but we're not there yet.

gaeldon Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

It also matters for thinking models and for agentic workflows, especially in software engineering, where a lot of tokens need to be output in iterative loops before the user sees any result.

This is our main use case.

gaeldon Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

In theory yes, although not in a linearly proportional way, because in practice our memory streaming is not yet perfect. There are still some fixed costs that we did not fully optimize (for now).

gaeldon Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

I'm sure there are, and I really hope we can work on consumer-grade GPUs at some point.

It should be possible to apply the same methodology (digging deep into the hardware details to understand all its little characteristics, and rethinking the inference stack around that).

gaeldon Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

Note that this coding model is trained on programming use cases, and is also not tuned for multi-turn chat.

You can ask it to implement an algorithm; we provide suggested prompts you can test.

Also, this tech preview is really about the speed of the inference engine (not the model itself) so I'm glad you got 3.4k tok/s!

gaeldon Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

Why not, it's one way to look at it! Although I have yet to see other work with speculative decoding higher than ~1,000 tokens/s., because the other bottlenecks start to matter at that point, and they need to be solved to go further.

Our view is that MTP / speculative decoding could help getting a X multiplier (X = 2 to 6) on the tokens per second speed we currently achieve.

We are a bit greedy, we want to stack optimizations on top of each other to get the maximum speed possible.

It involves additional compute to verify the predicted tokens during the forward pass (it's like a small batch), which should be totally doable for dense models, and will be more tricky for MoEs because it could mean activating more experts and thus more active parameters.

gaeldon Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

it's also a coding model

gaeldon Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

thank you deflator, I understand this now! much appreciated

gaeldon Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

Token generation speed matters for sequential agentic workflows, like software engineering / vibe coding, where a lot of reasoning tokens, code generation, refactoring, testing, etc. happen in a loop before an actual outcome is served to the user.

About model performance, we plan to support the latest frontier models (this tech preview is about the speed of the engine)

gaeldon Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

will do - we are a small team and it takes time to implement and optimize a new model, whatever the size.

gaeldon Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

Totally, though DTP is not required for these kind of speeds. Standard TP works also.

DTP is something we built for our roadmap in order to get to extremely high speeds (like 10k+ tokens/s). When the budget is under 10 µs per layer, any little overhead matters.

For 1k to 5k tokens/s, regular TP still works because we are able to optimize the inter-GPU all-reduce collectives at under 3 µs, which allows to continue streaming model weights in shared memory, registers and caches while GPUs exchange data.

gaeldon Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

Thanks for the comment and the question!

The last section of the article lays out the scaling laws that apply when porting this approach to another model. In a nutshell, DeepSeek V4 Pro with 49B active params is close to the upper bound.

Also worth noting that our results are currently for standard datacenter GPUs. On consumer hardware, though the same low-level optimization approach applies, the bandwidth limitations will cap the achievable speed.

gaeldon Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

Thanks a lot! Much appreciated.

To answer your questions:

- yes, we rewrite the whole model code (while keeping the same logic) in CUDA/HIP and assembly, in order to optimize by hand for each GPU type. It's quite tedious for sure, but I guess this is the price to pay to get this kind of results.

- the batching question is a great one. In agentic systems, there is probably a trade-off between sequential thinking/iterations vs parallel exploration of multiple solutions. Also, there could just be multiple independent tasks running in parallel, depending on the use case.

We plan to support a small amount of batching, but it quickly becomes a trade-off vs speed. Pick one for your use case, I guess.

Also to consider: because we answer requests much faster, we are also able to process lots of them without needing high batches - and scaling on multiple nodes is possible.

- open sourcing: maybe, maybe not. I'm still undecided on this. We are a small startup and I'm told that giving our IP away might be shooting ourselves in the feet. On the other side, I think it could be of great benefit to the community and for us... we'll see

gaeldon Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

Yeah, I agree: I'm actually not expecting it to be easy, and there will certainly be several unknown unknowns we'll discover along the way.

Our process has been, and will continue to be, a sequence of (tedious) R&D experiments where the GPU never behaves as expected when pushed to its limits in ways no-one really tested before (I still have nightmares of the L3 cache cross-IOD bottlenecks on MI300X).

IMHO, we did solve the multi-GPU memory bandwidth scaling problem, and thus the linear scaling of the size of the model towards infinity. But the main difficulties will come from keeping the speed, with steady and continuous memory streaming, while implementing the much more complex architecture of modern frontier MoEs (attention compression tricks, hash layers, routing logic, etc.)

gaeldon Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

Great points, let me clarify:

- model size: 2B is just for this preview (it was faster to implement), our article explains how we expect to support large frontier MoE at 1,000 to 5,000 tokens/s

- reaching 500 tok/s, or even up to ~1,000 tok/s, on a consumer GPU card is possible with existing inference engines like vLLM. But there is a ceiling.

The hard part comes we you try to be faster than that: these frameworks won't scale higher just by adding GPUs or using faster GPUs. There is a "glass ceiling" due to microseconds lost everywhere in the stack (grid syncs, inter-GPU comms, kernel launches, CPU sampling, etc.).

All our work at Kog is about removing these bottlenecks.

gaeldon Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

I updated the article title accordingly

gaeldon Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

YES - I just updated the title of our article according to your suggestion.

gaeldon Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

Follow-up reading the most technical and research people here:

Monokernel deep dive (GPU Engineering): http://blog.kog.ai/building-a-single-kernel-latency-optimize...

Delayed Tensor Parallelism (research): http://blog.kog.ai/delayed-tensor-parallelism-for-faster-tra...

To try the speed on the playground: http://playground.kog.ai

gaeldon Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

I guessed you thought about consumer GPUs. We are about standard datacenter GPUs indeed.

Sorry for the confusion

gaeldon Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

I guessed you thought about consumer GPUs. We are about standard datacenter GPUs indeed.

gaeldon Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

thanks! we explain how it scales to larger models in the last section the OP blog post

gaeldon Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

Thanks. To be fair, this number is what we expect to get once we port DeepSeek V4 in our engine on the upcoming generation of GPUs!

gaeldon Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

Great points.

We strived to be fair as possible in the benchmark, but it's indeed not perfect. Taalas should have been added in the dedicated hardware section, even though they use 3-bit quantization when we are on FP16 (to be fair in both directions) and they burn the model directly on the card.

Our tech preview is about the speed (hence the small dense model, it was easier to implement).

The math checks out though to allow support for large frontier MoE models at similar speeds: - At batch size 1, GPT-OSS-120B has 5.1B active parameters - in FP8, it's in the same size ballpark than our 2B model in FP16 (5.1 GB vs 4GB). - DeepSeek V4 Flash has 13B in mixed FP4/FP8, so let's say ballpark around 3x bigger than 4GB - so in theory we could reach >1,000 tok/s on it with MI300X/H200 and up to 4k on next generation GPUs.

Check out the math at the end of our blog post:

https://blog.kog.ai/real-time-llm-inference-on-standard-gpus...

gaeldon Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

Fair point - this tech preview is about the speed (hence the small dense model, it was easier to implement).

The math checks out though to allow support for large frontier MoE models at similar speeds.

At batch size 1, GPT-OSS-120B has 5.1B active parameters - in FP8, it's in the same size ballpark than our 2B model in FP16 (5.1 GB vs 4GB).

DeepSeek V4 Flash has 13B in mixed FP4/FP8.

Check out the math at the end of our blog post: https://blog.kog.ai/real-time-llm-inference-on-standard-gpus...