That is exactly how I think about it. It’s lossy compression. Think about how many petabytes of actual information any of these LLMs were trained on. Now look at the size of the resultant model. Its orders of magnitude smaller. It made it smaller by clipping the high frequency bits of some multi-billion dimension graph of knowledge. Same basic you do with other compression algorithms like JPEG or MP3.

These LLM’s are just lossy compression for knowledge. I think the sooner that “idea” gets surfaced people will find ways to train models with fixed pre-computed lookup tables of knowledge categories and association properties… basically taking a lot of the randomness out of the training process and getting more precise about what dimensions of knowledge and facts are embedded into the model.

… or something like that. But I don’t think this optimization will be driven by the large well funded tech companies. They are too invested in flushing money down the drain with more and more compute. Their huge budget blind them to other ways of doing the same thing with significantly less.

The future won’t be massive large language models. They’ll be “small language models” custom tuned to specific tasks. You’ll download or train a model that has incredible understanding of Rust and Django but won’t know a single thing about plate tectonics or apple pie recipes.