The technology, compression, etc, for models are getting better and better requiring less and less H/W and power to run the same inference power.
Perhaps our hunger for ever more powerful models will be enough to offset this, but I am not sure.
Will incredibly efficient new GPUs appear? will custom ASIC, models burned in silicons, take off? Will unified memory systems become usable?
The datacenter bets are that things will mostly continue like now, and that these GPUs won’t depreciate too much.
For now theses bets are self-fulfilling, causing hardware to even appreciate.
I used to think "wow, ADSL is so fast, I will never need faster internet connection" and now I need fiber to sync 500GB dataset fast enough so I don't get frustrated. Gosh, I remember the feeling when I upgraded from 14.4k to 56k dialup modem.
I expect as LLMs penetrate further, we'll see much more of it, continually running, and doing more 3D work.