Hacker News Reader

One of the main problems with LLMs today is they are gigantic, and this is because they are shipping the entire compressed memry of the training data with them.

Future LLMs are likely to have much smaller size and have outside long term memory/training knowledge as well as work memory (a'la RAG approach).

View on HN

Isn't this just ignoring all the lessons we learned from pretraining large models? Seems like a different flavor of the bitter lesson.

But in compressing all that data, they seem to build up internal representations that let them generalize on data they haven't seem. That reasoning ability, if not the actual data, is what gives them their power.

Reply on news.ycombinator.com