That argument swings both ways. There's no real reason to bet on OAI or Anthropic. Not with the rise of local models and the increasingly desperate attempts at regulatory capture.
It doesn't need to be the best. It just needs to be good enough. I'm not smart enough, but if I was I'd be working on a locally ran interface engine. That's where you charge say 100$ a year. Said engine would allow you to use a large open source model locally to the point where it's good enough to rival Haiku.
If I had to guess someone is working on this
Building the iPhone-platform equivalent of this would be interesting.
I'm far from an expert, but I don't think it's impossible. Future models won't give out vacation ideas, they'll focus on one or two things. This might allow us to get the size down.
Two observations - one, there is some set of tasks where intelligence is already saturated, and sure, that threshold is rising by the month. So, sure, for some tasks you don’t need the best model. But there are other tasks that need models smarter than what we have now - like “make a cure for <disease>” that will yield billions of dollars of value, but likely require quintillions of tokens and higher IQ/token than anything we have now.
As long as we have non-saturated economically valuable tasks there will be incentive to train new and smarter models, and the frontier will always be on racks of GPUs not local distilled versions.
Two, it’s not clear that you can distill all the way down to 32GB even for the current frontier models. IIRC Karpathy has speculated that you can get an AGI into GPT-3 size, if you drop all the memorized facts and keep the reasoning core. But this is highly speculative! The best 32GB models right now are substantially dumber than even the large open-weight models, let alone the frontier closed models.
Perhaps I am missing something, what are you imagining of “a locally ran inference engine” that doesn’t already exist in a DGX workstation, Framework desktop, MacBook, etc?
I think it's important to qualify that while I will very much appreciate such an innovation and I think it'll provide significant value to society... I remain doubtful of the ability for any company to actually capture such an astronomical yield. Either there's enough of an investment required by the company to help get that to market that the yield will be significantly lower or everyone and their brother will just launch competing products two days later. And, as a society, I don't think it's logical for someone writing "Hey Alex, cure cancer" with billions of dollars so I doubt there's going to be much appetite for creating such a moat via legislation.
This gets to the same point of "If ChatGPT can make you a successful million dollar business in a few prompts why is OpenAI sharing it" paradox. It very well may be that AI tools can accomplish the labor that used to be required to create such a business, and I'm sure a few people will land a jackpot and get rich in the transitional phase but if it's so cheap to build it why would someone pay you money for yours instead of just running the same prompting themselves? For most people that's because they lack the knowledge that they can but you'll still have a significant number of instant impersonators.
I agree fully, to be clear o was using that prompt as a shorthand, I don’t think _you_ can type that prompt and get $1B back.
It can’t do this now, obviously. You need a lot of scaffolding and domain expertise to get the most out of these models. (Eg Harvey.)
But eventually, the labs can just run the business themselves.
I’ve long held that as we see models become more capable, they will be less available to the public. It seems pretty obvious that vertical integration on private models (perhaps a per-industry fine-tune, eg a legal model) licensed to subsidiaries will let the labs capture more value.
Why would you publish your model, enable competitors, and enable distillation of your hard-won intelligence? Only antitrust law can prevent this outcome IMO, the business pressure is extremely strong.
Someone much smarter than me would figure this out.
One option would be an alternative bare metal OS which could allocate ram more carefully.
I'm sure with billions upon billions in VC funding a solution could be found.
Imagine if we had some kind of really intelligent AI model that we could pose problems like this to. It's obviously not the kind of thing an open weight model running on your local machine could do, but if we had some kind of really large, closed source model running in the cloud that could solve really difficult problems like this, I wonder if there'd be a market for it...
Regardless, looking back even a year at SOTA local models that fit on a non-absurd system creates a crazy trendline toward the future. And I think model routing is the gap filler that can make/break now vs. Oct 2027
I suspect Anthropic is of this general thinking as well; they've clearly switched from a thoughtful progression toward a natural IPO to "omg sprint for it". Along with the regulatory capture attempts, it's hard to imagine what is breathing down their neck so hard outside of this.
Totally agree local inference of any major scale is a nerd's wet dream short of many stacked improvements in efficiency. But again, auto-routing smooths over the rough edges if you are ok with a little bit of API spend when needed.