Fwiw - I wrote a paper documenting LLM (and neural) regression a few years ago (Neurips 2018)

https://arxiv.org/abs/1808.00508

They can interpolate but extrapolation is hard because of the constraints from nonlinearities