That hasn't been true for optimizing compilers either.

I thought that too, a year ago. But then chatGPT Turbo came out (ten times cheaper), and a slew of 30B and 10B models that are decent. Now I believe we will be able to run non-trivial AI on trivial hardware. Not to mention the Stable Diffusion revolution, hardware requirements went down pretty fast. Even an old GPU from 5 years ago can generate images quickly. Some LLMs run on iPhones.

The trick is always to offset the cost of inference with larger cost of training. You don't apply the Chinchilla scaling law, that's for academics and people who don't have to pay for inference. You pretrain the model 10x longer (like 1T tokens for LLaMA) to make it the best you can fit into an A100 or 4090 quantised to 4 or 3 bits. So everyone can have AI assistants running on their own toys.

10^16 -> 10^24 flops isn't 100x, it's 100,000,000x

This happened in 10 years (training cost of largest model according to the batshit crazy god AI article in the FT).

On the one hand compute power increased by about 100x and then some lunatics came and spent 10000x on it...