9,596 karma · joined July 28, 2014
Utter bullshit. We are still scaling transformer architectures initially introduced ten years ago. Ten years of phds trying to improve on what Google produced ten years ago, mostly failing.
What has actually changed and can explain the progress we’ve seen? Do you really think it’s just a collection of better and badder RL gyms? Get real!
The main thing driving progress in ai is massive amounts of capital investment and incremental improvements in hardware (mostly memory bandwidth/capacity) and computer networking (we got better at collectives). The AI labs, ironically, have nothing to do with it. They’re the vessel for capital. The people actually driving things forward mainly work at nvidia.
The reason the models are better today than they were 3 years ago is almost exclusively due to better hardware and infrastructure software. Not better data, not better model architecture. The evidence of this is pretty easy to feel: the reason opus 5 doesn’t feel much more capable than opus 4.6 did is because they run on the same hardware generation. The reason opus 4.6 felt much more capable than anything before it is because it coincided with the scale out of a new hardware generation.
In terms of cost per task, the open weight Chinese models are winning by a long shot. So it depends on what you mean by the best model.
This is a litmus test for the effective altruists in AI labs: fashion or conviction? The social pressure is to slow down and let the guild keep its norms a few more years. But what's clearly best for humanity is to push forward violently and leave them in the dust. A Fields medalist's feelings are not more important than the wellbeing of the planet.
By plausible deniability I simply meant adopting a similar approach that Stremio has, to keep it on the good side of App Store approval processes. The app itself only allows for creative commons content by default, but with the right incantations it can host anything.
Commenter below gets it absolutely correct: stockfish, which runs on your 5 year old phone, is dramatically better at chess than Fable. Like, so much better that it’s not even remotely comparable. The theory of the Bitter Lesson, and it’s only a theory, is that LLMs could eventually outperform stockfish. It’s not true today and it remains to be seen whether it will ever be true. For now, specialized models are absolutely better at specialized tasks.
I agree, but as a software engineer this gives me pause because I keep trying to insist on coding standards but I’m unable to come up with a compelling reason why it matters. Ostensibly the reason we cared about things like DRY and code quality was so that it would be easy to understand and easy to maintain and easy to make changes to later. But it now seems like a shared understanding of the codebase is less important than ever, and it’s more about shoveling requirements in without breaking any existing functionality.
Is a well tested slopfest better? That seems to be the conclusion for mathematics, so why not software too?
What? This is anecdotal and subjective. If the research doesn’t back up those anecdotes, then the research doesn’t back up those anecdotes. You want the research to back up your lived experience because it’s validating. That doesn’t mean that’s the actual reality.
The US is already publicizing the way they are using Claude with Palantir for war gaming purposes. It’s a matter of national defense. Contract law has no meaning here.
Do you really think intellectual property laws will prevent this in practice? It’s like as if we said, “hey, USSR, you can’t make a nuke, too! We patented that already.”
Asking China to not distill our models down is equally as ridiculous.
Even if it didn’t happen here, it was still the case that it was going to happen going forward. It was always going to end like this. Invest in the hardware companies, not the model companies.
I’m just surprised they need this many of them. What do they do all day? And who is left to manage their hundreds of thousands of servers?
It’s exceedingly unlikely that any of the people who were working on YC startups previously have any real professional experience with any of the following: slurm, collectives, NUMA systems, RDMA, compilers, systems programming, general HPC performance estimation or measurement, CUDA or ROCM or any kind of GPGPU/accelerated computing. But that is the core business of both of these companies.
I’m not surprised that these companies are well funded and hiring a lot of people. I’m surprised that they chose to hire the people who were previously making “Uber but for dogs” gimmick apps and not just hollowing out the HPC specialists from national labs.
What a ridiculous sentence. He’s an autocrat, but he’s out of power after losing a democratic election. Which is it?
Words have meaning.