You should only trust going into a library and reading stuff from microfilm. That's the only real way people should be learning.
/s
3,007 karma · joined October 23, 2012
You should only trust going into a library and reading stuff from microfilm. That's the only real way people should be learning.
/s
https://unexcitedneurons.substack.com/p/how-to-calculate-hom...
The (approximate) equation for milliseconds per token, is:
Time for token generation = (number of params active in the model)*(quantization size in bits)/8 bits*[(percent of active params in common weights)/(memory bandwidth of GPU) + (percent of active params in experts)/(memory bandwidth of system RAM)].
This equation ignores prefill (prompt processing) time. This assumes the CPU and GPU is fast enough compute-wise to do the math, and the bottleneck is memory bandwidth (this is usually true).
So for example, if you are running Kimi K2 (32b active params per token, 74% of those params are experts, 26% of those params are common params/shared expert) at Q4 quantization (4 bits per param), and have a 3090 gpu (935GB/sec) and an AMD Epyc 9005 cpu with 12 channel DDR5-6400 (614GB/sec memory bandwidth), then:
Time for token generation = (32b params)*(4bits/param)/8 bits*[(26%)/(935 GB/s) + (74%)/(614GB/sec)] = 23.73 ms/token or ~42 tokens/sec. https://www.wolframalpha.com/input?i=1+sec+%2F+%2816GB+*+%5B...
Notice how this equation explains how the second 3090 is pretty much useless. If you load up the common weights on the first 3090 (which is standard procedure), then the 2nd 3090 is just "fast memory" for some expert weights. If the quantized model is 256GB (rough estimate, I don't know the model size off the top of my head), and common weights are 11GB (this is true for Kimi K2, I don't know if it's true for Qwen3, but this is a decent rough estimate), then you have 245GB of "expert" weights. Yes, this is generally the correct ratio for MoE models, Deepseek R1 included. If you put 24GB of that 245GB on your second 3090, you have 935GB/sec speed on... 24/245 or ~10% of each token. In my Kimi K2 example above, you start off with 18.08ms per token spent reading the model from RAM, so even if your 24GB on your GPU was infinitely fast, it would still take... about 16ms per token reading from RAM. Or in total about 22ms/token, or in total 45 tokens/sec. That's with an infinitely fast 2nd GPU, you get a speedup of merely 3 tokens/sec.
Here's Deepseek R1 running off of RAM at 8tok/sec: https://www.youtube.com/watch?v=wKZHoGlllu4
https://arxiv.org/abs/2505.24832
LLMs usually have about 3.6 bits of data per parameter. You're losing a lot of information if quantized to 2 bits. 4 bit quants are the sweet spot where there's not much quality loss.
You will be mostly using 1 of your 3090s. The other one will be basically doing nothing. You CAN put the MoE weights on the 2nd 3090, but it's not going to speed up inference much, like <5% speedup. As in, if you lack a GPU, you'd be looking at <1 token/sec speeds depending on how fast your CPU does flops, and if you have a single 3090 you'd be doing 10tokens/ec, but with 2 3090s you'll still just be doing maybe 11tok/sec. These numbers are made up, but you get the idea.
Qwen3 Coder 480B is 261GB for IQ4_XS, 276GB for Q4_K_XL, so you'll be putting all the expert weights in RAM. That's why your RAM bandwidth is your limiting factor. I hope you're running off a workstation with dual cpus and 12 sticks of DDR5 RAM per CPU, which allows you to have 24 channel DDR5 RAM.
Spend $15 or $100 for one or two measurements, *then* worry about cost to mitigate.
For reference, a RTX 3090 has about 900GB/sec memory bandwidth, and a Mac Studio 512GB has 819GB/sec memory bandwidth.
So you just need a workstation with 8 channel DDR5 memory, and 8 sticks of RAM, and stick a 3090 GPU inside of it. Should be cheaper than $5000, for 512GB of DDR5-6400 that runs at a memory bandwidth of 409GB/sec, plus a RTX 3090.
The RAM is for the 400gb of experts.
https://learn.microsoft.com/en-us/azure/virtual-machines/siz...
A million determined voters can easily force laws to be made which forces youtube to be more efficient.
I often think about how orthodoxical all humans are. We never think about different paths outside of social norms.
- Modern western society has weakened support for mass action to the point where it is literally an unfathomable "black swan" perspective in public discourse.
- Spending a few million dollars on TV ads to get someone elected is a lot cheaper than whatever Bill Gates spends on NGOs, and for all the money he spent it seems like aid is getting cut off.
- Hiring or acting as a hitman to kill someone to achieve your goal is a lot cheaper than the other options above. It seems like this concept, for better or worse, is not quite in the public consciousness currently. The 1960s 1970s era of assassinations have truly gone and past.
By that standard, python is not real programming because you're not managing your own memory. Is python considered AI now?
If we somehow discover LLMs right after Newton discovered the theory of gravity, and then a while later Einstein discovers General Relativity, then GR would not be in the training set of the neural net. That doesn't make GR any less of a description of reality! You also can't convert General Relativity into whalesong!
But you CAN explain General Relativity in English, or in Chinese to a person in china. So the fact that we can create a mapping from the concept of General Relativity in the neural network of the brain of a human in the USA using english, to someone in china using chinese, to a ML model, is what makes it a shared statistical model of reality.
You also can't convert General Relativity to the language of "infant babble", does it make general relativity any less real?
The biological system of taxonomy is really for the biology of the organism. We have other categories we use (as humans who are not biologists), even though we borrow organizational structure from biologists! For example, the conceptual category of "vegetable" is a culinary term, not a biological one, and is a good example of a category not used by biologists.
There's a common saying, "Knowledge is knowing a tomato is a fruit. Wisdom is not putting it into a fruit salad". I believe that biologists should be categorizing organisms based on their origin, but people who are not biologists should not be bound by categories created by biologists.
For example, palm trees and bamboos are not trees biologically, but actually tall grass. The biological category of "tree" may not apply, but when you hire a landscraper, you aren't using the biological category of "tree", but rather the gardening category of "tree" (when you need a palm tree cut down). That's not a failure of biology, that's just because we use 1 word "tree" to describe 2 categories used by different fields.
Their Ampere A1 free tier is pretty good. 4 core ARM and 24gb ram webserver for free.
Claiming AI isn't unique in being a tool for evil isn't interesting, the point is that it's a force multiplier as such.
From someone who uses quotes „like this”?
Is there any big downside to do it the boring way, hardcode a list and compare the error to the list?
Right now, for a coding LLM on a Mac, the standard is Qwen 3 32b, which runs great on any M1 mac with 32gb memory or better. Qwen 3 235b is better, but fewer people have 128gb memory.
Anything smaller than 32b, you start seeing a big drop off in quality. Qwen 3 14b Q4_K_M is probably your best option at 16gb memory, but it's significantly worse in quality than 32b.
What do you mean by this?