In a lot of sci-fi games an upgrade path for ship/cybernetic hardware is to replace the 'AI chip'. If the models performance ever start to plateau I can't wait to see AI models being baked into the chip itself.
Llama 3.1 8B baked into an ASIC. Expect 1/10th the power requirements, and other savings.
This is one of the reasons I don't believe the "it'll never be profitable" narrative. Just stop training when you think your model's "good enough" and move to custom silicon.