> The “just by looking at the raw weights” setting (raw floats in context, no tools, no running the network) only appears in experiments 1, 2 and 4, on networks of 17K–172K weights.

The author is being equally disingenuous.

That piece by itself is insightful. Meaning, LLMs are capable of deciphering what a model can do by looking at the weights alone.

The fact that we can’t squeeze enough of them through the context is a technical limitation. However, this does not diminish the importance of this finding/discovery. This is an unexpected emergent behaviour.