anerli

How times have changed! Working on agents for so long helped us understand how inference needs to work to actually support local agents properly.

Hi, would you mind creating an issue on GitHub with information about what didn't work and maybe a screenshot?

https://github.com/magnitudedev/magnitude/issues

Currently it's not something our engine supports, but in theory our kernels could help for any training or fine-tuning jobs as well. If it's something you'd be interested in, feel free to create an issue on GitHub describing your use case!

Hey, yeah this is a known issue on M5+ macs. We are working on a patch so that our kernels use that hardware acceleration path. This should make prefill faster than MLX-based engines and boost decode a bit more for that hardware!

We have a couple bugs we are patching where we are reserving too much memory overhead, you should actually be able to fit a couple different models on there!

Plus in the near future, we'll add ways to automatically utilize your RAM as well (such as expert-streaming).

Our goal is to make the best use of whatever hardware you have, even if its not high end!

Yes :) same concept could apply there. If it's something you're interested in feel free to open an issue in our GitHub!

Hi, M5+ Macs have a known optimization gap since we do not fully utilize the Metal 4 matmul operations in our kernels yet - so should be able to do much better on this specific comparison soon!

Hi, the model ranking scores are based on whatever machine Magnitude is actually running on. So it will account for your specific memory capacity and performance characteristics to recommend appropriate models. The estimator is purely to help filter and recommend a model. The tuning is the only part that actually effects real performance, and is done automatically whenever you download a new model.

Yes MLX is generally a better comparison point overall for Apple, planning on releasing a benchmark for that soon. However against the MLX-based engines we've compared with so far Magnitude will continue to have an edge, especially for decode kernels.

Prefix cache is re-used with a prefix tree structure for maximal re-use across sessions sharing prompts.

The endpoint is standard OpenAI compatible chat completions.

Yeah that doesn't sound quite right. Given that the 5080 has 16 GB of VRAM I would have expected a few more options, for example Gemma 12B, to at least show up as available. Do any bigger models show as available or was that specifically the recommendation?

The MoE 35b might be tight though unless you were to go below 4-bit. Could you share the quant you used when you ran this on that 5080 before? Our catalog only contains models down to 4-bit because we find that thinking, tool calling, and overall capabilities start to suffer at lower fidelity.

Feel free also to create a GitHub issue with more details and we can take a closer look.

In the short term we will enable using multiple devices within one machine efficiently and distributing the workload of a model between them. If you mean running the same model distributed across different devices, yes we do have plans to enable that. You should eventually be able to run larger models across machines with capable hardware as long as you can provide a fast enough connection between them.

Core AI is a way to ship a model inside an app rather than an inference engine you can point agents to, and it's not meant for agent workloads.

Once you get down to the level of writing GPU kernels the OS isn't very much in the way, aside from some host-side memory logistics.

So it's more about making those GPU kernels perform the required memory move and arithmetic operations as close to the theoretical optimum as possible.

The goal of the inference engine is to make the best use of whatever hardware you have to run models performantly and let people run bigger models. At first this will include using all the hardware on a given machine optimally. Eventually we also want to support interconnect between multiple machines to enable running bigger models!

Will be adding this one very soon!

On our benchmarks we approach 2x decode speeds on a variety of Mac hardware (tested most on M4 Pro and Max).

llama.cpp does not saturate memory bandwidth for single-stream tok/s, and for long context and batching, our quantized KV and associated decode kernels allow us to reduce the effective bandwidth needed, and surpass llama.cpp significantly in decode speeds.

Thanks for pointing this out. I think we have a gap here where we may not be fully leveraging the new matmul operations available on M5+ chips, so will work that into our kernels soon and benchmark on M5 hardware.

These numbers are just estimates based on your hardware and may differ from actual performance. It's hard to get an accurate measurement until it's actually downloaded and running. They also don't account for gains from speculative decoding. Working on changes to make this more clear.

For m5 - there may be some issue with the Metal 4 matmul hardware utilization that could be causing this to be behind here. Will look into this.

While it's great to see tok/s go up as high as possible, I think it's important to consider the actual usability of these models when you quantize down to something like 2-bit. From what we've tested it seems like going below 4-bit quickly leads to serious issues with thinking, tool calls, and overall model coherence.

Our plan to enable running bigger models on less GPU memory in a way that'll remain productive is expert streaming. This will let you offload experts for MoE models to RAM or disk, and load them when needed. This can have some performance tradeoff, but is lossless.

Yeah we heavily leverage coding agents for optimizing our kernels. Since it's highly verifiable and takes time to measure we often leave multiple running and improving performance on different model architectures.

Definitely still helps to reference relevant academic work as well, or even just encouraging the agent to make bigger structural leaps, otherwise it will often get stuck working on low impact micro-optimizations.

No specific benchmarks for Strix Halo yet but planning to release more results for different hardware and models soon!

Qwen3.8-Flash-Next support will also be added very soon.

Taking full advantage of all the hardware on your machine in the most performant way possible is the overall goal of the inference engine. This includes a lot of what you're describing. We want to map out the full hardware topology of your system (one or more GPUs, CPU, memory), and compile a combination of kernels to serve a given model optimally across that stack, allocating different parts of the workload wherever it fits best.

Currently we're writing tunable kernels that optimize themselves for one device, but we're working on a kernel compiler that will be able to compile and distribute kernels across any number of devices in a system.

As mentioned here https://news.ycombinator.com/item?id=49912327 we'll eventually build an inference cloud for hybrid workloads. For now though, we're focused on making the inference engine great!

Thanks for reporting this issue - assessing is not supposed to take more than a minute or so. This is not strictly necessary but filters out models that don't fit in memory and gives speed estimates. This should ideally be a very short step so skipping hopefully wouldn't feel necessary if we patch this.

Could you share your hardware and OS details to help us identify what might be the issue here?

There's also a github issue open on this topic if you want to leave a comment there: https://github.com/magnitudedev/magnitude/issues/142

From the looks of it, this seems focused on datacenter/batch inference, and doesn't tune its kernels to the specific hardware and workload where inference is being run like Magnitude does.

Magnitude is optimized for maximum single-session performance and memory efficiency - so we should be more performant for local inference use cases.

Temperature control is a good callout! Feel free to open an issue for any features that you'd like to see in Magnitude

https://github.com/magnitudedev/magnitude/issues

Issues should already be open for anyone to report!

https://github.com/magnitudedev/magnitude/issues

Let me know if you keep running into problems for some reason

We are actively benchmarking our Vulkan kernels to ROCm implementations in other engines to ensure that we can reach the performance ceiling with them. Vulkan is much more portable and also works on non-AMD hardware even though it can be more awkward to write kernels for. If we find that Vulkan is not sufficient for reaching the same performance as ROCm, we'll consider adding it as a backend

I would say the overall idea of trying to achieve performant inference for agent workloads is the strongest commonality with Wafer.

It's not a coding agent running on your device optimizing the kernels, we have a system for writing kernels that can be tuned on the target device automatically. So we write the efficient high level kernel structure with tunable parameters, then it fits to whatever hardware it's actually running on.

We have a retrieval benchmark based on RULER which we've been using to ensure that the model maintains complete awareness of the full context window.

All our benchmarks are open source so you can check it out here if you'd like: https://github.com/magnitudedev/magnitude/blob/main/inferenc...

Right now, since we use less memory for KV, you have more room for model weights when you're running longer sessions.

However we also have expert streaming on the roadmap. This will let you run mixture-of-experts models with unused experts offloaded to RAM or disk, and load them only when needed. This means you'll be able to run models that wouldn't otherwise fit in your GPU memory.