No-one's going to want to talk to a 7B model for more than 3 minutes
Significant investment is being spent on producing hardware that is more suitable for running mid-sized (100B-ish) models. Starting with flash chips with wide enough interfaces to provide the weights (NAND flash can be ~5x more energy-efficient for reads than DRAM), and going into more exotic solutions like compute-in-memory.
My assumptions are that inference costs continue to decline (and/or the difficulty level of routing requests declines) and hardware acceleration continues into the future. These assumptions might be wrong, but that is the idea the thesis is based around.
(have built workflows and harnesses that currently target LLM provider APIs under the assumption I'll be able to point "locally-ish" eventually, my career had a strong focus on distributed computing about 10-15 years ago, think managing data and work clusters for LHC detector data processing, and I grew up on hobbyist distributed.net and BOINC projects, all the lenses I see this problem through; queues, jobs, sync vs async performance optimization, etc; recovered sysadmin|infra/devops engineer a lifetime ago)