But do you need to run every small problem through a 10B-30B model?
We're smashing ants with hammers most of the time. We're asking frontier Opus/Fable models to classify text and build frontend code.
Once we start dissecting these problems into smaller discreet tasks and having the big reasoning models do the tough stuff, we suddenly have an economical system. Not for the company hoping for a big IPO, but for the end user.
Then there might be slow, batch tasks. I can see myself getting 1T of slow RAM one day (in a few years?) and having a slow onsite GLM5.2 doing batch jobs that would be wasteful of my subscription limits, plus sensitive but boring things, such as bookeeping and general admin.
I'd like to to read all my email and al quarterly reporting. But that would have to be a good local model, probably a model simmilar to whatever google search uses, which seems just correct unless you throw serious challenges at it.
Actually probably yes: text analysis (magazine articles) by LLMs in the ~30b .. ~120b range failed miserably (and also randomly - the rare cases of proper interpretation occurred among the failure cases) with the main public models of around one year ago, tried extensively.
So, yes, you can employ an ~80IQ only if you will expect the related quality.
But I meant to counter gp’s claim that “I can run a 27b model on an iPhone” is kind of pointless and disingenuous. Yes I’m sure someone will come up with a way to run a “27b model” at 0.1 bit quantization on an Apple Watch pretty soon misses the whole point of saying a model is “27b” in capability.
Achieving a parameter count is not the point. And is almost meaningless