[flagged]
I mean how useful are GPT-3 weights today? That's the kind of useful they'll be relative to the frontier.
I feel like the so-called "Deepseek moment" broke too many people's brains and they've lost the ability to reason about why frontier advantages matter. I think we have to learn some new core concepts and rate of improvement over time is going to be a big one, and not understanding that causes people to exaggerate the significance of downstream distillation that's done in response to it. It's cheaper and open and that matters, but it's not the kind of mattering that replaces the relevance or advantage of the frontier, and I feel like I'm not seeing people demonstrate that they know how to really compare the relative merit of these two things so much as zoom in on one to talk past the other.
I was using GPT-3 and the original Llama when they came out and realized that it is interesting, but not good enough - yet. But they are good enough now. And just like there are still people driving Renaults and Volkswagens the latest batch of open models will be useful for a long time to come.
> whether you want to acknowledge that or not is not my affair.
I felt I acknowledged it pretty explicitly. I said the likes of Deepseek are "cheaper and open and that matters", but it's the rest of that sentence that I think went without acknowledgment.
>And just like there are still people driving Renaults and Volkswagens the latest batch of open models will be useful for a long time to come.
Walking is great too! But you're not filling an economic need by selling walking to the masses, and those companies are marginalized and the economic destiny of the world is being set by those who own the infrastructure to build out the world's next generation of electric vehicles (at least in the automotive space).
I think you're assuming that the options will be: "uber-super-ultra-premium intelligence" and then something like that but only slightly less so and much cheaper. Those aren't the only potential options. One other option is "uber-super-ultra-premium intelligence" and "nothing that really matters".
That seems to be the crux of the "open-weight" models versus "closed" models debate/discussion. I think it's more of a belief system being expressed than it is a reflection of any reality.
Open source software evangelists tend to have a quasi-religious viewpoint on the openness of software and that the model is superior and will always be cheaper and if not better only slightly worse, but because it's cheaper it's better. They will proclaim its universality, and apply it not just to web frameworks but to artificial intelligence, or state secrets, or anything else where there is some secret or knowledge to be held and maintained. If I were to say I think closed-source software is better and more preferable, open-source software evangelists will come out from all corners of the internet to make sure I know how impossible it is for security to be good on closed-source software, or how software yearns to be free or something. Then it turns into "the AI industry ... prefers to make claims about infinite exponential growth".
All of that is just a belief system. It's not scientific or factual. And it clouds judgement.
I don't know what you are comparing or how you are comparing things. You should provide some examples like, cost per token or some other useful metric other than "open-weight models are closer and closer".
Maybe the best way to think about it is like this: Yes a cheap Android phone from Wal-Mart has a camera, you can text on it, and install apps, but Apple makes all the money. Why is that? Because it's not good enough to just be a cheap and "close" alternative. If it were, then we'd see a race to the bottom on cost and commoditization, but we don't. Technology matters, but so does commercialization and it is immature and incorrect to think otherwise. If OpenAI and/or Anthropic go on to capture 90% of the market's profits and leave the bottom 10% for open-weight model companies or distilled models or what have you to fight over the scraps then OpenAI and Anthropic still win regardless of the capabilities of the other models.
Similarly we don't see this race to the bottom with closed-weight models. If open-weight is or was superior we'd just see nothing but American open-weight model companies come in to existence overnight and take over the market just like we do for any other software. But we don't. The fact is that open-weight models while providing some advantages are only one part of the market makeup.
You can think through this a bit too. First, you can see that China is now requiring export licenses for some of its AI companies and closing up models. Why is that?
Second, can you talk through the revenues of open-weight models? And if closed-weight model companies go bankrupt because of the supposed superiority of open-weight models where will these companies get their weights from and who will train the models and on what infrastructure?
It's like sure yea you can access the Internet and tinker around or whatever, and with some old LLM model you can have it do fun things, but on a scale of absolute usefulness and if we include what people actually pay money for and what is highly adopted, your laptop and model are severely outdated and by and large irrelevant. That's why the iPhone sells like hotcakes all day every day and twice on Sunday and Apple is a $5 trillion company. Society disagrees with your measure of usefulness.
Sorry I'm rate limited since Dang thinks I'm arguing to much on political topics (he's probably right), but to answer a couple of questions regarding "my old technology does this one thing" you are conflating your personal and nostalgic uses for old hardware/software with what is useful for society or business. If the old hardware was still just as useful as the new hardware it would be used instead of the new hardware.
Technology never was about the tech an sich but about what you do with the tech. For one person that 10 year old laptop is useless, for another it is a source of $500K / year in income. You, the user are the biggest factor in how much value you can extract from the technology at your command. I'm not sure how much longer that will be true but it definitely is still true today.
Compared to laptop quality, I suspect model quality will be a much bigger competitive advantage in the tournament of the marketplace. High-quality laptops are still limited by the quality of the software developer.
For now it is exactly the same with the models.
Also... employment in some parts of the stack is genuinely going away. I don't even look at CSS anymore.
To some degree, yes. But 'dwarfs' isn't really the term I would use. It is a bit slower, but the upsides are >> than the downsides for me. Not having Sam, Dario or Xi's minions looking over my shoulder matters to me more than any technological edge.
> Also... employment in some parts of the stack is genuinely going away. I don't even look at CSS anymore.
Yes, that's a fact, but that's orthogonal. I haven't touched either HTML or CSS in a long time now and yet I've cobbled together (I wouldn't claim ownership) a lot of stuff that 'just works'. I also would not expose that stuff to the world at large because I suspect it isn't nearly as secure as it looks.
I pay attention to frontend code just to keep code quality up. Frontend work is relatively low stakes from a security perspective and I've never seen the robots do anything problematic. I don't vibe code React but I think you could get away with it today.
Backend is a totally different story.
That's still a lot of engineering getting displaced.
The machine was bought in 2011 and it runs Windows Server.
I can do whatever I need with the device, apart from Gaming. Even YT works in 4k.
So, tell me now please why my 15 year old computer is not useful here?
My macbook is 5 yrs old and Im a big tech guy. Tech matures. It doesn't grow forever. You live in delululand bro.
Also, it's entirely plausible that an previously open model (that then became closed) will be much easier to distill than having a bunch of closed models.