Quadrillion-parameter transformers and how to train them | Hacker News Reader

Quadrillion-parameter transformers and how to train them

qlabs.sh

64 pointsfrozenseven60 comments

> Humans, by contrast, have ~100T parameters (counting each synapse as a weight) and learn from just ~200M tokens, a tokens per parameter of 1:500,000.

I don’t think that’s a fair comparison since (as far as I know) humans aren’t born with randomly initialised neural weights that are setup in a very generic structure, but have at the very least extremely strong inductive biases. I think it’s fair to count the hundreds of millions of years of evolution between the first animals with a primitive neural system and humans today, and the amount of “tokens” that were “trained on” is likely enormous. I think it’s better to consider the 200M tokens as a sort of “fine tuning” to a sort of genetic “pre-training”.

there's also the fact that the brain doesn't process inputs in discrete, feed-forward steps. everything is running continuously and feeding back to earlier layers.

and then there's the abstraction of 'activation' used in neuralnets that is very useful computationally, but omits a lot of the complexity in actual neural signals. you don't get fine-detail in the timing of spikes or the fact that neurons use different types of neurotransmitters.

That 200M figure needs much more justification as well, humans recieve huge amounts of data through their senses, particularly vision and touch.

A nice source for this might be "The unbearable slowness of being: Why do we live at 10 bits/s?" [1].

A lot of people seem to think that training is limited to the 10 bits/s of processing mentioned here. But raw sensor input (mostly visual) is in the order of 21 quadrillion bits per year [2], according to the same paper. It seems likely to me that learning uses much more data than is ultimately expressed in behaviour.

[1] https://www.sciencedirect.com/science/article/pii/S089662732...

[2] Assuming 16 waking hours, and 10^9 bits/s of input that the paper mentions. 16*3600*365*10^9 = 21.024*10^15

That "raw sensor input" is massively, hideously spatially and temporally redundant.

Even the shallow encoding patterns of AV1 can easily crush a raw video stream down by a factor of 10^3 without losing either spatial coherence or semantic loading.

Humans live in self-similar environments and perform self-similar activities. Any practical training dataset has the "natural human data" beat on diversity by a crushing amount. And a system that can learn deeper patterns makes the sensor streams more redundant, not less.

If it weren't massively redundant (ie identifiable patterns) there would have been no point evolving brains to parse the raw sensor input.

Learning something from a lecture, seminar or a book is close equivalent to learning from text/tokens.

Nah. The "huge amounts of data" humans receive are incredibly self-redundant.

Incorporating non-text data into AI training runs has repeatedly demonstrated that for general reasoning capabilities, no data type beats text in value per byte.

It is very redundant, but there is an incredible amount of it and we can "train" continuously and much more efficiently.

This is only true on the surface. Yes, making a model a good debator can be acchieved like this, but every current frontier model exhibits the same semantc confusion when it comes to actual mechanics of physical objects. Understanding how not to pass the ball through the glass is not a property of textual reasoning. This is also very visible in model generated code where the ball and the glass plane very often intersect. Easy to identify in syntax (compilers will do the job for you), easy to miss on the concept level.

This becomes more true the deeper you look.

LLMs used to "fail at spatial reasoning forever". But each new generation would fail a little less. Then we got the likes of Astra, and ARC-AGI-3 straight up died.

Our entire genome would fit on a CD, and much of that is used for metabolism and overall body morphology, not specifically brain structure; even though it almost certainly gives us a whole bunch of stuff (e.g. enough to recognise smiles and thereby allow them to form a positive reward we can learn from, and some approximation of how to sense a potential reproductive partner), it is absolutely not enough to give us a universal language nor intuition about any tool more advanced than a sharp rock and possibly the wheel and fire.

On the other hand, even such basics may well make a huge difference.

Analogy: a cryptic Sudoku puzzle whose only clues are the rules, vs. a normal Sudoku puzzle with a few clues.

I think it's the training set.

Pretty much every organism with a brain has a sophisticated survival reaction to physical contact with any part of its body, because its brain is trained on nerve signals. Everything with eyes or sonar learns spacial awareness of objects and their proximity, velocity, directoion of motion. I think the clue is that biological brains are trained on a physical world with very complex rules and physics as well as reactions that arise from chemistry.

Words and images, audio and video, even virtual worlds with soft-body physics, are just flat projected shadows of the complexity of the actual world.

> Words and images, audio and video, even virtual worlds with soft-body physics, are just flat projected shadows of the complexity of the actual world.

While true, so too are our senses: https://en.wikipedia.org/wiki/Aristotelian_physics

Edit: and conversely, look how mediocre sensory AI is. A trillion miles of driving experience over the Tesla fleet and reviews treat it like a young adult who only recently passed their test, the Cybercab is heavily geofenced, plus there's recent reports about them not correctly identifying the speed limit.

One time when I was a teenager I was outside with my house cat and she was doing normal housecat stuff like rolling around on the ground or sniffing stuff until suddenly for no apparent reason she went into a prone position with her limbs spread out and then bolted around in an odd zig-zag pattern and made her way to the corner of the garage overhang and jumped into the little opening that lead to the attic in the garage

I wondered wtf was going on so I looked around and realized that some sort of bird of prey was circling overhead that triggered that reaction and behaviour. It got me thinking. Somewhere inside of her brain and therefore her genome was the equivalent of a pixelated black and white image of the outline of a bird of prey, and the algorithm that can generate that zig-zag pattern in muscle firings and a whole overall heuristic that can apply these patterns and others into a coherent strategy that manifested as the fight or flight behaviour that I saw.

No one taught her this as she never really spent any time with her parents or her litter as she was adopted quite young, and spent most of her time doing house cat things so it wasn't something that she picked up from her environment, so it seemed pretty innate to me.

It was very influential to me as a teen to ponder that if that bird of prey's genome encodes the full shape and structure and behaviour of it then in some fractal sense there's a low fidelity copy of the shape of that bird and also implicity the bird's behaviour through a counter prey behaviour encoded in the cat's genome.

I recall that nature documentary showing how kangaroo fetuses climb up and into mom's pouch. They've been in a womb for the human equivalent of 7 weeks, yet immediately outside they know they must climb up, how to crawl through the fur, and how to identify when they've reached their destination.

I'm also reminded of that car AI using a neural net with a single shared weight, where the topology of the neural graph was what made the AI work.

How stuff is connected is what our DNA is about, so seems obvious the topology must encode behavior and such and it's not all learned.

Is this the line of “single shared weight” work that you’re referring to?

https://weightagnostic.github.io/ https://arxiv.org/pdf/1906.04358 [PDF]

Yep that's the one. Was in a bit of a hurry to dig up sources.

I agree with you. Also I really doubt we can so easily tokenize the input to the human brain.

If those tokens were to be passed on they'd have to be encoded somewhere, like DNA. And in DNA, there's no room. So ... where are they?

This is such a beautiful question. It looks like a miracle that a brain, let alone a whole body, can grow from such little data. Michael Levin has been doing a lot of research on this, you should check him out.

The thing is ... once those nerves are operational they bring in boatloads of data, which is useful and obviously tuned and very relevant to the specific body the neural network finds itself in.

No room - or is there?

FractalDB proves that "synthetic data useful for training AI" can be produced by a very small program.

There is "no room" in DNA for a full dataset, or a full snapshot of model weights. But there's plenty of room for "this one little trick cuts the data required to learn a desirable thing X in half" reproduced thousands of times for different X. Low Kolmogorov complexity, but high performance gains.

The data is massively compressed and pruned. Where are the input tokens in an LLM?

Well keep in mind we receive from our mothers an entire functional cell. DNA is a bit overrated.

As memes in the environment. Choices made by parents while raising us, things we hear from our peers, etc.

Also as aspects of nature... Its not just genes for a system that can feel pleasure or pain but also a mapping from that system to a set of textures, temperatures, and frequencies that take that system beyond certain thresholds. Plus a large set of environmental inputs that cause the system to grow in different ways in response to different stimuli.

DNA is probably the seed for the random initialization.

This is actually a genius idea. Imagine being able to compress a 1T parameter model to 10M params by creating a lossy model-scaffolding, like DNA

Something that more or less guarantees you’ll get about the same capabilities as the original model, but with a different final expression of certain characteristics

I wonder if this could be a good distribution model too, selling “dried” models (almost like raisins - or the beings from the 3 body problem), and then you can buy them to fully train (raise?) with your own data

Aside from violating everything we know about neural networks (technically this would mean there is a way to reduce the rank of even something like ChatGPT to 1. There isn't. No way) it would also be VERY unlike what DNA does. DNA has lots of ways to express things that aren't easily disturbed by mistakes. Those mistakes are then an actual positive, in the very long term, because they help natural selection along.

A random seed that works would be the exact opposite of that.

Why wouldn't DNA just encode "good feelings" when IT gets what it wants? Right temperature -> ahhh. Food -> ahhhhhh. Sex -> aaaaaahhhhhhhhhh. And then just let reinforcement learning take care of all the details in between. No need for storage space for fire making or for "Hey girl, how are you doing?" ... and therefore cheaper.

> Aside from violating everything we know about neural networks

Not sure exactly what this means

But there’s prior art on the concept of compressing models through lossy representations

In particular, procedural generation of models, check out HyperNEAT (2009), HyperNetworks (2017) and Developmental Symbolic Encoding (2011)

I'm somewhat of a genius (self-described.)

I like your idea of "base" models for sale.

It's nice to throw around claims like that, or like "microtubules connect the brain the quantum realm" (Penrose). Fortunately that's not how science works.

I understand what happened. You took my claim literally. Like DNA translates by a simple function to random initialization. I was saying that all else being equal, DNA will influence the way the brain initializes. Of course, many factors will influence it.

Science starts with a hypothesis. If DNA has everything encoded, then it stands to reason that it influences the initial state of the brain.

> If DNA has everything encoded

It is not. There are many biological examples of random processes generating patterns and weights, such tiger stripes, fingerprints, etc. DNA only encodes the reaction, but the pattern itself is seeded by initial position of random molecules, not the DNA.

> that it influences the initial state of the brain.

You've gone from "DNA is probably the seed for the random initialization", as in the deterministic seed for a PRNG, to "influences". Those are 2 different claims.

Most scientists will agree that DNA encodes the large-scale connectivity in the brain, ie from region to region, thickness of corpus callosum etc.

Most scientists will disagree that it is "probable" that DNA encodes/seeds the weights of each and every synapse.

I think you are mostly debating against a strawman of your own construction here. The person you're responding to never claimed that "seeding the random initialization" means literally the same thing as seeding a pseudo-random number generator, and the second message was likely intended as a clarification after being misunderstood. So it's not "2 different claims" contradicting; it's a single underlying claim being expressed through a lossy medium in an increasing level of detail.

My own reading of the poster's intent aligns with this: "DNA encodes the large-scale connectivity in the brain, ie from region to region, thickness of corpus callosum etc."

The grandparent comment is: "humans aren’t born with randomly initialised neural weights", with commenters throwing around harebrained hypotheses about those weights.

Most biologists will fully disagree with the whole premise, and support the reverse claim that we are born with random weights, we learn to walk/speak English/etc after we're born. Not from weights seeded from our DNA.

Except Penrose is like Chomsky. They have past accomplishments, but nothing recent and everybody keeps bringing them up because they constantly comment on stuff so far outside of their area of expertise it's not funny anymore, it's just sad.

At least Penrose had the good manners to make relevant contributions (meaning Chomsky effectively managed to steer his field in entirely the wrong direction, take everybody along with nothing to show for it. Nobody uses his contributions today in study. Yes he's celebrated ... but why? He's like a prime minister that presided over a country causing an economic crisis, throwing everyone into misery. I guess those are exactly the kind of politicians celebrated. Bush is celebrated while he destroyed half the economy, Clinton is forgotten (well, except for that one thing) despite being the best president of the last 50+ years economically)

Most scientists think Penrose's theory of microtubules is wrong, however he is still far better than many consciousness theorists in giving an actual testable criterion.

Tokens being passed on would be what, you close your eyes and experience your ancestors lives?

No. More like, your ancestors that have a fight or flight response survive more often and have passed that down.

Or that infants with a rooting response survive much more often.

Basically, weights encoded biologically from hereditary behavior.

Instincts.

Now, the real conundrum is that if our DNA can really only encode ~12gb then how are we passing this information down?

biological life is a continuous networked system. watch avatar. lol.

You sound just like Noam Chomsky when it comes to this :) He always talked about the LAD!

The human brain is nothing like an LLM. The truth is that LLMs are incredibly inefficient and actual AGI will be created from something completely different.

> Our new paper (Baron et al., 2026) finds that when the dataset is held fixed, the compute-optimal model size keeps growing with the compute budget.

The dataset is not held fixed in practice. It would be interesting to see an ablation study of where best to allocate compute between more synthetic data, more RL rollouts, and this paper's suggestion of cranking up the parameter count. Unfortunately that's an experiment only a frontier lap could accurately do, and they are unlikely to provide such expensive and valuable information to their competitors.

> The dataset is not held fixed in practice

The dataset is held fixed here because it's a control variable, and control variables are how you isolate relationships in science experiments.

The planet has 20 quadrillions of ants

And several thousand quadrillion grains of sand.

Edit: "several thousand" could be in the order of millions! (i.e. billions of quadrillions of grains of sand.) Apparently the number of grains of sand per this planet is not so clear. At least to me.

There are more ways to shuffle a deck of cards than there are atoms in the visible universe.

There are more ways to configure a single atom than there are card shuffles of a single deck.

what do you mean configure? I guess there are a lot of different electron orbitals but not quadrillions

I could have sworn there were infinite (but as energies increase they become increasingly easy to ionize).

Maybe for some theoretical ideal but definitely not in practice.

A typical grain of sand contains in the order of hundreds of quadrillions of atoms. A deck of cards can be encoded in 226 bits per deck, so one could encode in the order of a quadrillion decks of cards in a single grain of sand!

What if it's a MtG deck?

Yes synapses are weights, you know science, you smart

Why not? Its obviously not an accurate enough model for many things, but its a step in the right direction for making theories of the brain short of a full molecular simulation.

I don't understand the utility of this kind of thinking.

If it is wrong then it is wrong, why base more things on top of it if it is wrong?

It doesn't bring anything useful to the discussion, it is just low iq slop. Why is it so difficult to compare on real parameters like energy, time etc?

This is the same sort of insanity that we have in databases where people compare databases by the amount of milliseconds it takes to execute some query.

Why include it if it is useless, why just not do it?

Why use newtonian or lagrangian mechanics when we know its wrong? It was a important step, obviously far more correct than other nonsense interpretations of brain and consciousness that are still popular eg souls or magic. Aside from the biological inspiration, as a mathematical model it has proved its value beyond all expectations as well.

Paper: https://arxiv.org/abs/2610.37842

Reply on news.ycombinator.com