It's a fairly safe bet that the same would apply to LLMs. If you start with a simple uncompressed LLMs that is 65B parameters and somehow compress or quantise it down to less than that, it will inevitably become a little dumber.
But if you compared the raw LLM to one that utilised all of these tricks and was the same size, then the latter would be superior because it could use the available parameters more efficiently.
If we can train and run GPT-3 cost effectively now with ~100B parameters, then it's a safe bet that we could train something as smart as GPT-4, with >200K window sizes, but as fast as GPT-3 for inference. (That's assuming all the recent quantization techniques are also applied.)
I'm betting we'll have something like that generally available within two years.
That'll be terrifying. An AI that can read and understand a book every few seconds...