>Finding that 70% of attention heads and 20% of feed-forward networks can be excised with minimal effect on in-context learning suggests that large language models are undertrained.

I thought that it was common knowledge that LLMs are undertrained, none of the publicly available loss graphs show any sign of convergence!