It's going the other way. The number of levers to pull around architecture, data amount/type/quality, hardware optimizations, decoding optimizations, evals, system architecture (distinct from model architecture), alignment tricks and various other things means the complexity is just getting started.
Because of scaling advantages, it likely never will be. Even if you could run a GPT4 quality model on your laptop (you can't), by the time you can, Open Ai will be running a model that uses orders of magnitude more compute.
Stupid question, but why aren't they doing it already? What's their bottleneck right now?
Try shoving a couple trill (estimate) parameters in memory. Or, shuttling it back and forth from disk as required to generate a single token. And then shuttling that into the chip. Assuming they are using FP16 it's 4 TB give or take. They aren't kidding when they call them "Large".
I dont think compute capacity will be the determining factor for moat. It will most likely be access to various types of data and I believe on that front Meta and google are way ahwad
This could easily end up like the PC revolution though. ChatGPT being the mainframe of AI, and when the hardware catches up and performance is “good enough” we’re all running our own models all over the place.