I don't think training sets will be a problem. All that data the big houses have is going to leak, and while things are more and more locked down, we've seen some interesting copyright decisions out there.
One interesting concept I've mused at, is that certain countries (China for example) have immense public data to train off of, and a state which can simply say "You must make this data available to train AI". That's a large chunk of data right there, and if anything might cause Western democracies to loosen copyright law and slap some fix for training, that might be it.
"If we don't take action, the best models will be Chinese!", with panicked overtones. This could certainly hold some truth, if models cannot train newer data.
--
In terms of token cost, in 5 years every desktop and laptop will be able to run well beyond Opus5.5 compute. Even high end phones will be able to do Opus5.5.
We've seen a lot of improvements in open model space (MoE was one interesting concept), but beyond that, look at how just baking certain instruction sets, eg mmx, sse, avx in silicon helped immensely with certain compute tasks. Or look at the specific immense speedup of an fpu. Conjoin some change such as that, along with general compute improvements, and unified memory, and you're going somewhere for local in 5 years.
I've been in this industry decades. And at every single step, people have said "But it's too expensive to do $thing", or "There's no way to handle that level of compute". If you bet against vast improvements in this field?
You're going to lose that bet. So I do think yes, 100%, improvement, in some capacity, will be there.
Maybe this time is magically different. I just don't think so. I think in 5 years, there will still be frontier labs, and an API to call, but that in-home compute will be as I said. Able to run immensely capable Opus5.5 class or greater.
That's going to be an astonishing market shift, even just for in-home.
And if you think of it, I would be very surprised if some for of home robotics wasn't very common at that point in time. We can argue all we want, but androidian robotics is very good at navigating human spaces, everything from stairs to weird closets and everything in between.
And with the immense, almost panic level of concern about our aging population, and who will take care of the elderly, robotic assistance are at the top. Japan is betting big, and if you look at Opus5.5, if it had a little more compute, it could do 30fps video for navigation.
We're not there, but we're VERY close. And if we do get that degree of robotics, every robot at rest becomes general LLM compute. If such a robot simply closes its eyes and goes to rest, eg to charge, or at night, it now has enormous spare compute to use for other tasks.
Nothing, of course, is a given. But if you look over the last 2 years, and then take even a fraction of that progress and change over the next 2 years, the concept of massive layoffs isn't hard to imagine.
Case in point? I have great breadth and depth of knowledge in this field. I've been able to produce what would take a team of managed 4 junior engineers 6 months to do, in literally under a month. Me, and a $200+ LLM monthly buy. Absolutely the project would not have worked without my experience and directing, reviewing, and correcting, but I'm now acting as a manager, and replacing $1M/year in engineering cost.
With $200.
That degree of astonishing change is not going to sit idle without repercussions. Most especially, if there is any more improvement.
If we see another Opus 5 -> 5.5 level improvement happen, or better, if we seen two improvements such as that? I believe my statements about your (and my) careers will be entirely accurate.
--
To speak to my narrative, it said 'AGI', not "LLM", and it was along a decades long path. I do think that modern LLMs give great strength to exploring other research paths, far more rapidly. And the big houses have loads of spare compute they can throw at R&D for tests, so I think iteration down unknown but hopeful paths will move faster and faster as time goes on.
I fully agree that current LLMs are not on that path. There are interesting projects to modify weights and add knowledge dynamically as bolt-ons, but it's unclear if they'll just be... hacky, or integral in the end.
We came a long way with x86/32bit, then hacked on 64bit afterwards. Now ARM is muscling into mainstream, but it's still just the same old compute, hence my agreeing that such bolt ons won't change just improve what is.
Still, my general point as I've just said was that if AGI happens (and again I didn't say LLM based AGI), being quiet is the ultimate way to just wait.
When I started in this field, I worked in assembly, I worked with eprom burners, I played with Vic20s, C64s, and the level of experience and understanding I have is vastly improved via that tract. I understand computing differently, and in more depth in some ways, than people that didn't have to learn these things. And I had the luck of learning on very simple systems, where you could keep the entire platform in your head as you worked on assembly, without even referencing documentation.
That time is gone, and the field is immense now.
Think of how there used to be simply 'Physicists', but now there are 100 sub-branches and specialties in that field. 100+ years ago, on person could work in the entirety of that field. Not now, and the same is true of compute.
I guess what I'm saying here is, if you've seen what I've seen, you see a steady path of less generalist, and more targeted understanding of compute, and in each case a reduction in the overall comprehension of the entire field by specialists. Each laye of abstraction tends to reduce comprehension.
And back to the start, this means that I do see massive layoffs, and more and more abstraction, so that eventually no one will handle code directly.
Which means no one will ever know how anything works any more.