Neural networks are clean

His launch post gives good context not found in the blog:

> interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had

> with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works

https://x.com/nickcammarata/status/2110757819427574056?s=46

But isn't the point that the neural nets learn some insanely abstract ideas anyways? So whether the complexity is "ahh we think this neuron does 100 seemingly unrelated things but we don't understand it" or "this actually is some kind of latent shared feature, but we don't know what the concept represents because the composition of the entire circuit is an arcane algorithm", we're kind of stuck, no?

This seems like a classic model-description issue more than anything.

Once sequenced, the thinking goes, we'll be able to do things like: explain and anticipate failures; predict what happens when you modulate the pathway an input takes on its way to becoming output; reconstruct algorithms the models have learned.

I'm not 100% sure this will work out like the mechinterp people think. If the neural representations and pathways are "clean" (organized into a a meaningful-to-humans hierarchy), it seems at least very plausible that the behaviors that AI xrisk people are worried about will mostly boil down to straightforward synthesis (i.e., mostly-linear-interpolation) of mostly-well-cataloged knowledge. And if that is true, then it does not seem hopeful that there would be a way to prevent the output they are worried about, other than just removing a bunch of the knowledge. So maybe there is no alignment solution at all.

https://already-clean.goodfire.com/microscope/medium.html?la...

This type of error may have mislead researchers for years; it was actually harmful. In my field, a correction, or perhaps retraction (depending on the centrality of the claim), would be standard for something of this magnitude.

I suppose there are different standards in different scholarly communities. And maybe it truly was impossible to know at the time that the dog/car relation was actually a V shape, but my recollection is that this type of feature was well known at the time and so the error could have (should have?) been caught a decade ago.

So: massive new discovery, or a spin on a correction that's lead scholarly research astray for almost a decade?

And emergence doesn't need to involve much of the LLM's weights:

> In preliminary audits of Qwen3-32B, we estimate that roughly 90% of neuron poles have one main job, often with small related offshoots; another ~7% have two related jobs; and only ~2-4% are not yet understood

If those ~2-4% are the central hubs of emergent cognition, then understanding 96%-98% of the weights may be less valuable than the author thinks.

The idea is: a model may only have 100M neurons or whatever. But there are infinite concepts in the world. Therefore, models have to "stuff" multiple concepts per neuron.

How do you do that? Theres two ways:

1. Each neuron handles similar concepts. One neuron for dogs, a separate neuron one for cars, etc.

2. Each neuron handles a bundle of unrelated concepts. One neuron handles dogs and cars. Another neuron handles cats and numbers. And so on.

Historically, people thought #2 was correct because many neurons were found that seemingly represented multiple concepts. #2 had a cool name (polysemanticity) and was promoted in OpenAI/Anthropic blogs so it had a lot of traction.

However, this blog says #2 is wrong. Actually, the dog/car neuron is just a "V shape neuron". So #2 was just methodological error, and #1 was right all along. This is all the blog is saying.

https://x.com/allTheYud/status/2110805233454731449?s=20

For example, a trillion-parameter transformer model built using many layers is likely significantly easier to read than one that only has a few million parameters but was constructed using neuroevolution over one gigantic layer.

The degree to which a model can be quantized without losing its performance also likely contributes to interpretability. The redundancy can help humans lock onto patterns. If there is no redundancy at all (everything is a surprise in information theoretical terms), then the stream of parameters will look indistinguishable from cosmic background radiation.

"The move from a structuralist account in which capital is understood to structure social relations in relatively homologous ways to a view of hegemony in which power relations are subject to repetition, convergence, and rearticulation brought the question of temporality into the thinking of structure, and marked a shift from a form of Althusserian theory that takes structural totalities as theoretical objects to one in which the insights into the contingent possibility of structure inaugurate a renewed conception of hegemony as bound up with the contingent sites and strategies of the rearticulation of power."

https://www.theguardian.com/books/1999/dec/24/news

> The more we read [the neural network] the way its own readers do, the more interpretable I think it will look. We should be wary of reading it too much like a human, and concluding it’s messy just because it doesn’t fit our preconceived concepts.

Nick is looking at a 32B model, which certainly has to compress less... Now we have 10 trillion parameter models... do they need to compress at all? As far as I can tell, experiments show that Fable has basically memorized every lat/long position on the map of the earth can can tell you at a glance whether your standing in the ocean or not based on your coordinates

This approach is part of a way some researchers in the field approach interp, (me including) because I am personally cautious of developing some new representation model that claims to be the way we should represent the network. It is another layer of abstraction that takes us away from core model components, very often this requires training a separate model that was not on its own a part of original model organism during training (SAE).

I would argue that models will likely be further increasing in complexity, not just in size, when model trains continuously and becomes adaptive yet parameter count does not change, it is not hard to imagine that neurons may reorganize themselves to pack more information.

I always catch on misuse of enormity

It's now so widespread the misuse has gained despairing acceptance, as with "infer" for "imply"... but that doesn't make it any less wince-inducing.

If its grouping circuits based on motifs and when they fire — and it applies to large networks — then one should be able to literally query or debug a reconstructed database and check against actual activations. More importantly If this was a hand-wavy thing to do earlier for safety, it's not anymore because it's on more solid ground using Normal Science.

I especially enjoyed this subtle joke: "We think the weak activations usually don’t matter, but the strong positives and negatives matter a lot. For instance, Santa lives on the negative pole."

I can only imagine how much joy he felt writing that.

As for serious comments on its content, I'm confused by the people who considered it difficult to read. I think the ideas presented were very elegant and fairly simple to understand. Considering neurons as gates and weights as votes was a nice way to look at it, for example. I didn't feel like the post was unnecessarily long either.

One thing that stands out to me is how ridiculous it is in the first place to try to assign a short, natural language description to the function of any particular neuron. Wittgenstein already talked about how humans seem to use words in very specific ways while being terrible at exhaustively describing why.

An example is the word "game". It's very easy for even a child to distinguish between "game" and "not a game". But it's incredibly difficult to write down a precise definition of "game" that includes everything normally considered a game and excludes everything that isn't considered a game. Writing down the function of one specific neuron seems to run into this issue. When you write down what the exact function of any neuron is, there's a good chance your clunky natural language description misses some fine nuances.

A reasonable analogy, in my view, would be that neurons could represent something like the atomic components of intuitions or "vibes". Similar to the example of "game", the feeling of knowing what is or isn't a game is composed of what Wittgenstein called "family resemblances". Your brain doesn't contain a precise description of "game", but upon encountering something, you can quickly compare it to other examples of games you've seen and intuit whether something has a "game-like vibe".

Another notorious example would be "what is a woman". For some reason, everybody has very strong feelings about what is and isn't a woman, but people have a terribly difficult time perfectly describing it. I would guess that neuron activations function like these vibes or intuitions, rather than written rules.

I very much liked the parts discussing circuit ecology. This idea sounds very promising for alignment to me.

Two eyes, a nose with whiskers [0], a mouth and a protruding shape, four legs and a tail [1].

A car and a cat might as well be the same thing.

[0] the logo on the radiator surrounded by horizontal stripes [1] tow hitch.

Perhaps our anthropomorphic understanding of data is biasing what objective features there actually are in the dataset.

I wonder if we can study the graph similarities of various models, do we see a convergence of structures that encapsulate certain concepts. And would that convergence be due to just how humans have narrowly defined the concept in our language, or is there something inherent to the concept that allows it to converge. If we compare the same concept in different languages (ie. chinese vs english), does this convergence disappear?

The good news I guess is the actual patterns make a lot of sense with what you'd expect the neurons to do, and don't require nearly as much 'black magic who knows what wonders scale can create'.

> Part of the reason was not knowing about generalized pose neurons

I have to ask, did they not look up how the human visual system works? The primary visual cortex has any number of early neurons that are spatially-tuned for various shapes and edges, and this is pretty basic knowledge.

In fairness, neural networks are more divergent from biology than a lot of people think, but it seems there's still low-hanging fruit to be grabbed with a minimal collaboration.

Even when similar weight activations arise across learning regimen it seems we will miss a whole range of interpretations if we just go off 'the text'. The outside context in which a neuron arises is equally, if not more, telling; its genesis cannot (yet?) be quite captured by the weight itself.

The network may reveal its own context and history, but what can ultimately be encoded in its weights that originated outside the network?

https://pastebin.com/raw/p11vzFnB

If the field of neural net legibility develops tools for identifying plagiarism, that would be great. If we can trace output as derived work of certain specimens of training data, we can then clamp down on all the stealing.

If those results hold, it is a real breakthrough in the sense that it is a sharp simplification of an actually complicated problem.

It's interesting to observe that human reasoning is still the barometer of what "reasonably adjacent" means in the context of a NN's structure.

https://xxcancel.com/NeelNanda5/status/2110875974296654062#m (and the re-post on LessWrong: https://www.lesswrong.com/posts/x4kvabkF2Jk8GLZp8/linkpost-n...)

TL;DR:

    This is interesting work, but I don't think it's a significant update to the big picture of interpretability. [...] I'm on board with the broad idea of doing exploratory interpretability with neurons, e.g. neuron-based attribution graphs, for pragmatic problems like why did a model do a sketchy thing? I think this work provides some reasonable evidence, in addition to good earlier work from Transluce, that this might actually work for practical purposes, and has some advantages over the standard alternatives like SAEs/transcoders. I do not agree with some people's grander inferences like that this is substantial progress towards ambitious reverse engineering, solving interpretability etc.

There are no neurons or anything like them in these image and language models.

You work in data science and you over indexed on a singular example to explain away an entire set of phenomenon??

My experience with more.... directly testable NN models that are small seems to be that they absolutely can be far too interconnected to make strong explanations. The structure and pathways they fall into don't have to make inherent sense to anything or anyone.

Is it possible that almost all nodes in a network being explainable with a singular feature that makes obvious sense to us is evidence of poor efficiency? The network hasn't been made to collapse learning or discover more effective organizational structures than what might make sense to us.

I don't think I have the fundamentals here to contribute though.

Working with some neural networks, you can somewhat feel the structure when you read between the lines of the output. You can sometimes feel that a threshold somewhere tripped and set the signals down an alternative path.

But confirmation bias and all that.

I hope interpretability has a breakthrough. It would make my job far easier and cooler.

I had figured the "Recursive generalization" was like, understood? Maybe I'm just too pop-sci brained. "Split categories into more on this new boundary" and "Combine two separate categories that aren't different enough" I thought were always the core verbs for working with bulk data. My mental model has never been formalized though, so is invalid.

I guess I never got a good sense of the examples they had which made it look like what was "obvious" or made sense to me wasn't always a good explanation in general. Neat.

It's interesting that meaningful examples both ways care about shapes so much, as your eyes encode edge finding and shapes really early in the signal path. It's why some optical illusions are so effective. It's awesome.

The whole "how do you test a neuron: Find a group of nouns and find where you can split them" is just how I find I think learning works. You build your mental model, and find the split points where it diverges from reality and you are wrong, and you refine. I feel like this is how I learned Random Forests conceptually, though surely that's not a valid comparison?

Apologies, not sure I have actually contributed anything here! I bet I misunderstood the entire article somehow. Specifically, I do not have the fundamentals of looking into visual models that he is really working from. But I always figured that when Data Scientists talk of "Features" in larger models, they explicitly meant these far more abstract representational "features". Doesn't the premise of model hijacking and abbliteration sort of presuppose that you have abstract "Reject" or "Safety" neuron paths? I've always understood this stuff as that neuron path, without weighting too much on individual neurons.

>I used to think of neurons as human-shaped nouns: one is a car, another is a curve.

I didn't think anyone was ever supposed to think this for human neurons! The way humans identify something as even a "Chair" is insanely abstract and cultural! Very specifically, the "building sub branches to disambiguate" is just, like, how you would expect a hierarchical human brain to self organize, right?

Some of this is why I've been so displeased with the interpretability research from Anthropic, where it felt like they just pointed new NNs at the disparate neurons of another NN and post hoc named them. I felt the paths were of importance at least but had no idea how to convey, interpret, or model that. Learn you a good math kids! Not having any formalisms hurts.

What's next? Maybe in a little bit we will discover that we H. Sapiens suffer from severe polysemanticity in our wetware because our individual neurons are messy and massive like an ENIAC while the corvids in the park have neat and efficient little brains that don't need unfolding ?... How's the pope (may God stop afflicting him with cancer or may Man make him live forever) going to explain that?

Also how it goes is this new consensus will be a leaky abstraction and we'll keep conjuring new words to describe new rhetorical features and bloat the corpus of philosophy until everyone is over the whole concept and then we'll start over with a whole new arbitrary academic philosophy for "running algebra on a computer"

...and it still doesn't answer it. What is this actually about?

If you want to learn more about this, this is what depth psychology is about (Carl Jung, et al).

This is also essentially the idea behind monotheism vs polytheism. It's about engaging with and trying to understand the profound underlying unity of the myriad phenomena.