In fact I bet you could update like one or two neurons for certain concepts, and then transplant those neurons to another LLM to give it some idea of it. Like a literal brain transplant but for concepts.
Drop a set of neurons and there’s no change? Probably doesn’t contain the “sky color” concept.
Drop a set of neurons and the model freaks out, definitely conceptual neurons.
Rinse and repeat to find the distilled pattern across all the neurons.
You could train an LLM against the neuron graph to do this for you.
From a physics perspective it’s entropy: There are just more local minima that have neurons code multiple things.
I suspect that dropout and similar tricks also play a role in this: Removing connections during training means that pathways need to be redundant somewhat.