The trick is always to offset the cost of inference with larger cost of training. You don't apply the Chinchilla scaling law, that's for academics and people who don't have to pay for inference. You pretrain the model 10x longer (like 1T tokens for LLaMA) to make it the best you can fit into an A100 or 4090 quantised to 4 or 3 bits. So everyone can have AI assistants running on their own toys.
This happened in 10 years (training cost of largest model according to the batshit crazy god AI article in the FT).
On the one hand compute power increased by about 100x and then some lunatics came and spent 10000x on it...