Beautiful piece of math.
Similar work: Traveling Words: A Geometric Interpretation of Transformers https://arxiv.org/abs/2309.07315
That was an interesting abstract, looking forward to read it!
Would this kind of understanding aid in extracting 'skills' from an LLM by identifying the relevant topology and isolating that part? Then we could have a toolbox of skills to assemble into what we needed?
Maybe the right title would be a (statistical) physics perspective...
Disappointed that this wasn't about robots in disguise.