I am not sure you're correct. Yes, from the physical perspective of LLMs design, the above is correct. But they are also systems that evolved to do things through training; so you cannot discount the possibility they can emulate building of new abstractions in another level.

LLMs already have the concept of "reference" - i.e. I can say a name of something, for example of a function, and they understand that the properties associated with the name itself are not relevant, but rather whatever the name does refer to.

But this is pretty much what you need to build new abstractions. You need to name a paragraph with the definition (explanation) of a concept, and be able to recall the paragraph by the name in order to do inference.

So with references, they can get around the context size limitations, like humans can. It's slower than learning things "by heart", but I think LLMs are perfectly capable of that; and the evidence is, they are able to navigate codebases larger than their context, which often contain many new abstractions.