HNHacker News
TopNewBestAskShowJobs

jychang

3,007 karma · joined October 23, 2012

submissionscomments
jychang··on Study mode
Yeah, this is why wikipedia is not a good resource and nobody should use it. Also why google is not a good resource, anybody can make a website.

You should only trust going into a library and reading stuff from microfilm. That's the only real way people should be learning.

/s

jychang··on AMD CEO sees chips from TSMC's US plant costing 5%-20% more
I doubt that, unless you're willing to pay USA workers Taiwan salaries.
jychang··on Qwen3-Coder: Agentic coding in the world
Inspired me to write this, since it seems like most people don't understand how fast models run:

https://unexcitedneurons.substack.com/p/how-to-calculate-hom...

jychang··on Qwen3-Coder: Agentic coding in the world
How many channels of DDR4 ram? What speed is it running at? DDR4-3200?

The (approximate) equation for milliseconds per token, is:

Time for token generation = (number of params active in the model)*(quantization size in bits)/8 bits*[(percent of active params in common weights)/(memory bandwidth of GPU) + (percent of active params in experts)/(memory bandwidth of system RAM)].

This equation ignores prefill (prompt processing) time. This assumes the CPU and GPU is fast enough compute-wise to do the math, and the bottleneck is memory bandwidth (this is usually true).

So for example, if you are running Kimi K2 (32b active params per token, 74% of those params are experts, 26% of those params are common params/shared expert) at Q4 quantization (4 bits per param), and have a 3090 gpu (935GB/sec) and an AMD Epyc 9005 cpu with 12 channel DDR5-6400 (614GB/sec memory bandwidth), then:

Time for token generation = (32b params)*(4bits/param)/8 bits*[(26%)/(935 GB/s) + (74%)/(614GB/sec)] = 23.73 ms/token or ~42 tokens/sec. https://www.wolframalpha.com/input?i=1+sec+%2F+%2816GB+*+%5B...

Notice how this equation explains how the second 3090 is pretty much useless. If you load up the common weights on the first 3090 (which is standard procedure), then the 2nd 3090 is just "fast memory" for some expert weights. If the quantized model is 256GB (rough estimate, I don't know the model size off the top of my head), and common weights are 11GB (this is true for Kimi K2, I don't know if it's true for Qwen3, but this is a decent rough estimate), then you have 245GB of "expert" weights. Yes, this is generally the correct ratio for MoE models, Deepseek R1 included. If you put 24GB of that 245GB on your second 3090, you have 935GB/sec speed on... 24/245 or ~10% of each token. In my Kimi K2 example above, you start off with 18.08ms per token spent reading the model from RAM, so even if your 24GB on your GPU was infinitely fast, it would still take... about 16ms per token reading from RAM. Or in total about 22ms/token, or in total 45 tokens/sec. That's with an infinitely fast 2nd GPU, you get a speedup of merely 3 tokens/sec.

jychang··on Qwen3-Coder: Agentic coding in the world
What cluster do you have to do the quantizing? I'm guessing you're not using a single machine with a 3090 in your garage.
jychang··on Qwen3-Coder: Agentic coding in the world
ChatGPT is wrong.

Here's Deepseek R1 running off of RAM at 8tok/sec: https://www.youtube.com/watch?v=wKZHoGlllu4

jychang··on Qwen3-Coder: Agentic coding in the world
In the current era of MoE models, the system RAM memory bandwidth determines your speed more than the GPU does.
jychang··on Qwen3-Coder: Agentic coding in the world
Speculative decoding doesn't change output tokens.
jychang··on Qwen3-Coder: Agentic coding in the world
You definitely want to use 4bit quants at minimum.

https://arxiv.org/abs/2505.24832

LLMs usually have about 3.6 bits of data per parameter. You're losing a lot of information if quantized to 2 bits. 4 bit quants are the sweet spot where there's not much quality loss.

jychang··on Qwen3-Coder: Agentic coding in the world
How fast you run this model will strongly depend on if you have DDR4 or DDR5 ram.

You will be mostly using 1 of your 3090s. The other one will be basically doing nothing. You CAN put the MoE weights on the 2nd 3090, but it's not going to speed up inference much, like <5% speedup. As in, if you lack a GPU, you'd be looking at <1 token/sec speeds depending on how fast your CPU does flops, and if you have a single 3090 you'd be doing 10tokens/ec, but with 2 3090s you'll still just be doing maybe 11tok/sec. These numbers are made up, but you get the idea.

Qwen3 Coder 480B is 261GB for IQ4_XS, 276GB for Q4_K_XL, so you'll be putting all the expert weights in RAM. That's why your RAM bandwidth is your limiting factor. I hope you're running off a workstation with dual cpus and 12 sticks of DDR5 RAM per CPU, which allows you to have 24 channel DDR5 RAM.

jychang··on Many lung cancers are now in nonsmokers
Yeah, this is the correct heuristic.

Spend $15 or $100 for one or two measurements, *then* worry about cost to mitigate.

jychang··on Qwen3-Coder: Agentic coding in the world
They have similar memory bandwidth compared to the Mac Studio. You can run it off CPU at the same speed.
jychang··on Qwen3-Coder: Agentic coding in the world
If you're running it for personal use, you don't need to put all of it onto GPU vram. Cheap DDR5 ram is fine. You just need a GPU in the system to do compute for the prompt processing and to hold the common tensors that run for every token.

For reference, a RTX 3090 has about 900GB/sec memory bandwidth, and a Mac Studio 512GB has 819GB/sec memory bandwidth.

So you just need a workstation with 8 channel DDR5 memory, and 8 sticks of RAM, and stick a 3090 GPU inside of it. Should be cheaper than $5000, for 512GB of DDR5-6400 that runs at a memory bandwidth of 409GB/sec, plus a RTX 3090.

jychang··on Qwen3-Coder: Agentic coding in the world
Qwen uses 16bit, Kimi and Deepseek uses FP8.
jychang··on Qwen3-Coder: Agentic coding in the world
Well, yeah, you're supposed to put in a GPU. It's a MoE model, the common tensors should be on the GPU, which also does prompt processing.

The RAM is for the 400gb of experts.

jychang··on Qwen3-Coder: Agentic coding in the world
You can get similar performance on an Azure HX vm:

https://learn.microsoft.com/en-us/azure/virtual-machines/siz...

jychang··on A 14kb page can load much faster than a 15kb page (2022)
A million people is a very strong political force.

A million determined voters can easily force laws to be made which forces youtube to be more efficient.

I often think about how orthodoxical all humans are. We never think about different paths outside of social norms.

- Modern western society has weakened support for mass action to the point where it is literally an unfathomable "black swan" perspective in public discourse.

- Spending a few million dollars on TV ads to get someone elected is a lot cheaper than whatever Bill Gates spends on NGOs, and for all the money he spent it seems like aid is getting cut off.

- Hiring or acting as a hitman to kill someone to achieve your goal is a lot cheaper than the other options above. It seems like this concept, for better or worse, is not quite in the public consciousness currently. The 1960s 1970s era of assassinations have truly gone and past.

jychang··on My favorite use-case for AI is writing logs
Yeah. This is a really weird complaint to be honest.

By that standard, python is not real programming because you're not managing your own memory. Is python considered AI now?

jychang··on All AI models might be the same
I fail to see how that matters. You're implying that all reality is cultural, but that seems irrelevant. The same thing would apply to scientific facts, but whales not having a word for science doesn't make it not real.

If we somehow discover LLMs right after Newton discovered the theory of gravity, and then a while later Einstein discovers General Relativity, then GR would not be in the training set of the neural net. That doesn't make GR any less of a description of reality! You also can't convert General Relativity into whalesong!

But you CAN explain General Relativity in English, or in Chinese to a person in china. So the fact that we can create a mapping from the concept of General Relativity in the neural network of the brain of a human in the USA using english, to someone in china using chinese, to a ML model, is what makes it a shared statistical model of reality.

You also can't convert General Relativity to the language of "infant babble", does it make general relativity any less real?

jychang··on Show HN: Pangolin – Open source alternative to Cloudflare Tunnels
You want Tailscale for that.
jychang··on There's no such thing as a tree (phylogenetically) (2021)
Ehhhh I strongly disagree with your statement.

The biological system of taxonomy is really for the biology of the organism. We have other categories we use (as humans who are not biologists), even though we borrow organizational structure from biologists! For example, the conceptual category of "vegetable" is a culinary term, not a biological one, and is a good example of a category not used by biologists.

There's a common saying, "Knowledge is knowing a tomato is a fruit. Wisdom is not putting it into a fruit salad". I believe that biologists should be categorizing organisms based on their origin, but people who are not biologists should not be bound by categories created by biologists.

For example, palm trees and bamboos are not trees biologically, but actually tall grass. The biological category of "tree" may not apply, but when you hire a landscraper, you aren't using the biological category of "tree", but rather the gardening category of "tree" (when you need a palm tree cut down). That's not a failure of biology, that's just because we use 1 word "tree" to describe 2 categories used by different fields.

jychang··on Bought myself an Ampere Altra system
Oracle OCI?

Their Ampere A1 free tier is pretty good. 4 core ARM and 24gb ram webserver for free.

jychang··on My AI skeptic friends are all nuts
Yeah, that's exactly completely missing the point. A bayonet multiplies a person's power by 1.1x, a nuke multiplies it by more than 1,000,000x. Trying to be cute and lumping them together as "every technology is a force multiplier" is peak autistic literalism of 1.1x and 1,000,000x both technically being a multiplier even if they're clearly different.
jychang··on My AI skeptic friends are all nuts
Well, duh. Same thing applies for "Technology X can be used for war". But anyone with a brain can see nukes are on a different level than bayonets.

Claiming AI isn't unique in being a tool for evil isn't interesting, the point is that it's a force multiplier as such.

jychang··on Bus stops here: Shanghai lets riders design their own routes
Uh, "we" ?

From someone who uses quotes „like this”?

... https://i.imgur.com/swpYbpv.png

jychang··on Ty: A fast Python type checker and language server
Nontrivial way to do it is dynamically scan the python 3.12 namespace, and add these warnings.

Is there any big downside to do it the boring way, hardcode a list and compare the error to the list?

jychang··on Mistral ships Le Chat – enterprise AI assistant that can run on prem
16GB on a mac with unified memory is too small for good coding models. Anything on that machine is severely compromised. Maybe in ~1 year we will see better models that fit in ~8gb vram, but not yet.

Right now, for a coding LLM on a Mac, the standard is Qwen 3 32b, which runs great on any M1 mac with 32gb memory or better. Qwen 3 235b is better, but fewer people have 128gb memory.

Anything smaller than 32b, you start seeing a big drop off in quality. Qwen 3 14b Q4_K_M is probably your best option at 16gb memory, but it's significantly worse in quality than 32b.

jychang··on Claude's system prompt is over 24k tokens with tools
> Combined with VS Code’s ‘# %%’-demarcated notebook cells in .py files

What do you mean by this?

jychang··on OpenAI reaches agreement to buy Windsurf for $3B
But they don't OFFER Openrouter a paid product... You cannot give roo/cline dollars and get openrouter api access.
jychang··on OpenAI reaches agreement to buy Windsurf for $3B
Roo/Cline doesn't offer Openrouter, markup or not.
← PreviousPage 10 of 28Next →