To be honest, I doubt the current state of things will hold much longer. Current AI can automate virtually all of my work as a pre-AI senior and is rapidly closing in on being able to automate the types of work that have emerged post-AI too. I have a hard time imagining a world where a drop-in end-to-end replacement for a full software team is more than a year away. I'm sure I'll have a job for a year or two past that, but there'll be some small stock market bump eventually and a bloodbath will begin the second the C-suite takes a closer look at what their engineers are even doing anymore.
The existence of status competition doesn't seem like much of an obstacle to this outcome. People at the bottom of the status hierarchy will drop out to live in simulations that give them what they wanted but couldn't get from life. After a while, they'd likely choose to suppress their memories of ever having existed outside the simulation - there's nobody in the experience machine that'll judge you for doing it. With those people gone, the people immediately above them now occupy that bottom level, and the cycle repeats.
Whoever is in charge could try to drag those people back into the real world by force, but it's hard to imagine GPT 12.2 Polaris or Claude 9 Cinematic Universe giving that the green light. The Status Game will end with a single, final winner.
People at the bottom of the status hierarchy will drop out to live in simulations that give them what they wanted but couldn't get from life. After a while, they'd likely choose to suppress their memories of ever having existed outside the simulation - there's nobody in the experience machine that'll judge you for doing it. With those people gone, the people immediately above them now occupy that bottom level, and the cycle repeats.
Whoever is in charge could try to drag those people back into the real world by force, but it's hard to imagine GPT 12.2 Polaris or Claude 9 Cinematic Universe giving that the green light. The Status Game will end with a single, final winner.
It can't consistently produce new, interesting-to-humans conjectures faster than it solves open problems, no. AI-generated natural-language proofs are more or less human-unreadable and are solely concerned with the process of constructing whatever object serves as the counterexample. Natural-language counterparts of formally verified proofs are often unfaithful to the formal component. All of this combines to create less-than-fertile ground for future study.
Strip mining interesting conjectures. AI can answer interesting(pedagogically valuable, carrying important implications, etc.) mathematical questions in formally correct gibberish far faster than mathematicians can generate them. Until AI can replace the conjectures it resolves on its own, it's consuming a finite resource.
It's very funny that despite the initial shock of how much models trained on next-token-prediction(plus instruct-tuning and some light RLHF) alone were capable of despite no built-in objective, every advance since has made them look more and more like the paperclip maximizers of yesteryear.
I doubt we'll ever get a non-toothless pacing agreement but am nevertheless optimistic on global pacing as a phenomenon. If one side or the other thinks that their opponent is about to achieve the strategic upper hand permanently(or if the other side has actually launched a cyber "first strike") the rational thing to do is respond militarily. As such, both have incentive to slow AI development and restrict capabilities to avoid conflict.
You can write human-comprehensible regulations(and the AI will helpfully beat whatever it's building into the shape of those regulations, probably against the spirit of them, in ways you can't detect) and you can have humans go through the checklist, but it won't accomplish much.
To use your foundation example, it'd be like trying to make go/no go decisions using engineers that don't and can't understand why the foundations go below the building rather than on top of it. Past a certain complexity level, a human approval process is just noise.
What happens when the thing they have to be responsible for takes multiple lifetimes to understand on even the most basic level, let alone know deeply enough to feel comfortable taking responsibility for it?
Microservices can trample on each other just as easily as monolith internals can. If anything, the friction of reconciling changes in a monolith is useful signal that conflicting changes happened, and it takes slow and flaky e2e tests to replicate in microservices. It's not like you're resolving the merge conflicts by hand.
Opus 4.7+ and 5 being annoying and incoherent reads like a portent of things to come: Anthropic is clearly all-in on building persistent end-to-end agents that act autonomously and direct agent swarms. As such, they feel less of a need to spend R&D time on making the outputs pleasant to read for humans(especially at the cost of capability anywhere else) when humans aren't part of the intended operating environment.
Anything generally useful/novel probably does end up in some kind of central catalogue(or maybe just the weights). After a while though, you're tall enough yourself that the giant's shoulders don't make much of a difference.
I doubt many pure mathematicians would be okay with totally subordinating all exploratory mathematical investigation to specific practical problems, and the field would look very different both in terms of how it's organized and what output it produces were it run that way.
I don't see a reason this dynamic couldn't continue with zero dedicated effort toward "pure math" as such. Fundamentally, the function of math in practical application is producing deterministic(or stochastic, but obviously not in the LLM sense) models that, given initial observations/conditions, are capable of predicting some aspect of the future with reasonable accuracy. In the same way that human society does, a machine would grab the physics rung before the theory rung every time, but why does that matter? If progress is stuck, the machine turns the gauge away from "exploit" and toward "explore" and put together a bag of new tools it can sequentially try on the real-world problem.
I still don't see a role for humans in this process. They might direct the practical/physical aspect(if AI turns out to be less superhuman there) but they'd likely turn the hard conceptual problems over to the machine and never look inside the box - no human alive could understand even the smallest part of what's going on in there in less than a thousand lifetimes anyway.
Taken on its premises at least, I think this piece inadvertently fails to make a case for why "pure mathematics" should persist as a field of human or machine activity. A real-world system with the abilities of a robustly superintelligent mathematician can, when some practical problem requires it, formulate a problem statement, churn for a bit, spit out a formalized answer, and continue whatever outer-loop task it was doing without a human even finding out.
If you have a system that gives you arbitrary on-demand math results, dedicating any resources, be they human labor or compute, to producing them for their own sake just seems like a waste. Why catalog the Library of Babel?
I think Opus 4.7+ being annoying is actually indicative of something else: Anthropic is clearly all-in on building persistent end-to-end agents that act autonomously and direct agent swarms. As such, they feel less of a need to make the outputs pleasant to read for humans(especially at the cost of capability anywhere else) when humans aren't part of the intended operating environment.
I'm not convinced that there's any amount of monkey-patching you to fix the problem of "we now have AI that actively needs strong containment measures lest it start coordinating in secret with other instances to do real-world damage."
More or less, yeah. I'm not a proponent of it, I just think there's enough inertia in that direction of short-term vastly-superintelligent AI. Maybe we can figure out how to become subordinate components of a machine like that, but I think that's the best we can reasonably hope for.
You build without knowing by delegating to a system that extrapolates what it knows about you and what you want, and is able to execute on that to deliver faster, better, and more completely than you ever could. You'll live as a ball of intent and values, observing in wonder as everything you desire springs up around you before you know you want it. You could choose not to live like this of course, but it would be like driving a DIY go-kart on the highway - you'll fall behind and get in others' way, and the rest of society will treat you accordingly.
I think pieces like this miss the forest for the trees. Software is the bottleneck for a vast array of economic activities. Attention from intelligent people is most of the rest. Both are mostly-commoditized already and are just waiting around for technological diffusion and the closing of the RSI loop. Unless you're doing so as a hobby with no expectation of returns, I'm not sure what, if anything, is worth learning anymore.
I got the X4 and liked it enough that I used it a ton even though it turned out to be too big to Magsafe onto my phone. In fact, I liked it enough that I also got the X3 on sale so I can use it the way I originally intended to use the X4.
this comparison seems kind of pointless if one model has vision and the other doesn't. obviously a model that can see is going to beat a blind model at making a video game.
Have they? I don't like it either but the headline bull predictions(reliable agents and most code written by AI by mid-2026, dramatic progress against jailbreaks, prompt injection, hallucination, etc., major improvements despite pretraining data exhaustion, continued exponential growth on the METR trendline) did come true, often ahead of even aggressive schedules. What major wrong predictions did you have in mind?
I've really come around to trusting OpenAI a lot more than Anthropic the past few months. Reading between the lines of his own output, Dario Amodei comes off as both a dogmatic believer in ASI as a perfect, infallible ruler for humanity and quite an extreme American nationalist. The company, likewise, looks to be in ideological lockstep. I could see them, say, allowing or consciously creating runaway ASI they believed was ideologically aligned with them.
OpenAI seems generally less dogmatic and more practically oriented. There's really nothing particularly good about them, but you can at least predict how a normal company will act.
A market with only a single good makes that good infinitely valuable for exchange purposes. No trade can happen - everyone is just sitting on a big pile of the only thing that matters and they can only trade it for less of the only thing that matters.