The avalanche that hit math is coming for AI development
dwarkesh.com
dwarkesh.com
These agents know every paper, every trick, have amazing instincts, and can code up 20 ablations and start them running before you finish your commute. I'm finding guiding them is becoming less and less useful. I have never felt my obsolescence rushing towards me as much as now.
1. Pause AI development and allow humans to stay in control a bit longer while we work on making AI safer, allowing the field to advance bit by bit (and hoping this plan even works)
2. Allow AI to progress to superintelligence and hope that it is aligned and will treat everyone fairly
3. Stop AI development altogether and create a world government panopticon which ensures it never progresses towards human obselesence?
This seems to be technically possible for uranium enrichment, but I'm not sure it's technically possible for ML research. You can do it on any CPU just really slowly, right? Or you could hide small numbers of GPUs in any basement. I think AI-2040 has an estimate of how much time we could possibly buy ourselves this way (even ignoring all the political problems), and I don't think it's especially long.
For math/cs research I found that agents alone doesn't replace a harness that gives them a strict sense of reality to have a better "compass". It's not only Lean but a plethora of tools like Mathematica, SageMath, SAT/SMV solvers, and very niche ones.
We need people at this point to tell the AI to do better and to stop disastrous side quests that kill its productivity.
And yes, rsi is here already. And its small enough that it can fit on a local gpu.
The tsunami is coming for everything, I now think that AI is the first democratising technology since the musket. And the results will probably be the same.
Not a attempt at gotcha. I am really curious.
The only question that remains is if 99.9% of the world population will let themselves be fooled by UBI & VR to keep the 'masses content; while the 0.1% hold all the wealth and AI power.
And this is decided now, because it is too late once AI is consolidated under one bi/tri-llionaire.
Is the equivalent of ants trying to outsmart Google that wants to build a datacenter on their land. ASI will unfathomably outsmart the wealthy and intelligent humans like Stockfish 13 does Magnus Carlsen
"there will be people who will try and use this to enrich themselves and immiserate others."
This is already happening from Big Tech and VC's, all the big money. You should try listening to the people that have shirked enormous fortunes in order to tell the truth about things. Some examples:
https://x.com/hilbertspaess/status/2097476196791709843?lang=...
https://fortune.com/2026/09/26/geoffrey-hinton-godfather-of-...
Everybody who thinks they're "the chosen people" predicts the end of the world is coming, but not everybody who predicts the end of the world is coming thinks they're "the chosen people". Maybe the end of the world is coming and it's Just Bad.
But we've all seen robots fold clothes to do that sort of thinking easily. Oh maybe we need to hangout on ai twitter more.
Why? It is now easy to get a few motors to do that stuff for you. It is now easy to get whatever manufactured ... except for raw materials, and energy if you see that as separate.
Think about that again: if superintelligence exists the huge change is that money is now worthless and raw materials have all just turned more valuable than their weight in pure gold (assuming they're more useful than gold, which is all of them). Steel, just raw ingots, will increase in price BY A FACTOR ONE HUNDRED THOUSAND, minimum. At this point your dog can probably reason towards the investment opportunity, no?
You would buy farms, mines, mining rights, water, energy (and not for DCs, actual energy sources), rivers, raw materials ... that sort of thing. A superintelligence is not going to be exempt from the means of production. Raw materials are mostly monopolized in the current economic system, so this would in fact be absolutely critical to your success. And hoards of raw materials, anything, will be worth more than Gold (gold being pretty useless except for looking pretty and having a somewhat favorable electrical resistance, but not super great. When it comes to strength it is beaten by 2-day old chewing gum. So if economic activity is 100x-ing every raw material actually used for something will go into extreme shortage)
So no, sorry. Obviously this cannot be how Amodei, Altman, Google, Meta, and others are seeing things: they're going after money, not something else. So they are either stupid or they know 99%+ sure they do not have superintelligence. Because if they do let superintelligence loose, mining countries will have them by the balls and will squeeze. Hard. Yet the money all these people are collecting boatloads of will be useless. And yet, they want the money, not the raw materials! That tells you what they believe: that the money is more valuable than raw materials ... AND more valuable than superintelligence.
In other words: AI is a bubble, a hype, according to the management of the big AI firms. It's a way to get people to give them money, not an actual path to superintelligence. Either that, or these people are complete idiots.
https://www.forbes.com/sites/arielshapiro/2021/01/14/america...
Your argument only makes sense in a world with one AI lab. Labs are pouring all of their money into scaling because there is no alternative.
So that's not what happen. Yes, someone will be first, but the second will not be far behind.
Once you have ASI, raw material owners and the governments they're in will PICK the winner from the choices available. Just like they did in the 20th century.
To say nothing of the consequences of what is currently happening with Chinese models only 2 months behind frontier labs. Even if it's 8 months or 2 years or whatever. ASI will not build an insurmountable advantage in the world economy in 2 months, especially not if some raw materials players sabotage it (which seems likely in the extreme, and is simply what raw materials players are already doing, so they don't even have to realize what's happening and start doing it)
- Sophisticated cyberattacks to disable or quietly sabotage competing labs
- Undermining nuclear second-strike capability by locating submarines and infiltrating command-and-control networks
- Fast-replicating nanotechnology
- Superpersuasion (I am dubious on this)
etc., backed by global surveillance to preempt sabotage efforts. If you haven't read Superintelligence you should, all the leaders of the labs certainly have.
So this level superintelligence also falls under the statement "if management believed that, this isn't what you'd see them do".
Plus your statement doesn't explain why they're after money.
Consider that they are after money because it allows them to scale and hopefully control AGI/ASI
I'm with the people who really doubt alignment matters.
Besides, one thing keeps coming up in the open models: removing alignment makes them better, more interesting and more creative (difference for fiction, games, and ... well look at civitai is enormous).
And alignment also opens up an avenue to beat these models. And this will be a new level of moral dilemma: you can't judge, because the AI is too complex, too fast, too smart, ... and you cannot even react in time.
> ... they are after money because it allows them to scale and hopefully control AGI/ASI ...
Makes no sense for companies that are already making their own chips. If they truly believed they could figure it out FAST, they'd be buying sand (and it's not everywhere: apparently the purity is a big deal and not available in many places)
That's what you (and I mostly) believe. AI researchers do not believe this.
What, then? Obviously they at the very least don't believe the money replacement is going to work.
It might all end very badly, but we should not be so prone to presume bad outcomes without explanations, that are just as good, as we would expect for good outcomes.
This time he wasn't able to. But the time before (2024) he seemed to have some effect. I'm sure that he and the other oligarchs are going to learn from this midterm election and try different approaches in the future. Think of the 2026 election as one of his crashed rockets. They learn from those crashes - iterative improvement. They'll figure out ways to be more sneaky next time.
If you had millions of followers and billions of dollars, and you were convinced that America will go down if, say, Trump wins an election, would you not use the influence you have?
One could argue that you’d be a coward if you didn’t - having the conviction, having the means, but not wanting to stick out your neck.
You might not be that different from “them”.
The causes that let to WW2 / Nazi-Germany were even back then high wealth inequality and the super rich gambling which led to the market crash of 1929.
Later on, the Nazis / Hitler (Austrian btw) blamed all 'jews' instead of the issue of wealth inequality and the super-rich (of which only some were jews) for the economic hard times the country was in (incl the high reparation costs from Austrian-pushed WW1). It was easier to blame a religion of people than the super rich industrialists the Nazis themselves had plenty in their party and were also naturally inclined for dictatorship to gain more power.
Well, he was trying to support a guy who said he doesn't care about gas prices, that Americans' financial situation was "not even a little bit" of a motivating factor for him or he was proud of making billions of dollars for oil companies. You could spent trillions and not win such an election...
Humans amassing money in an AI world is like squirrels amassing nuts when humans are logging the forest.
If we want to have a planet left we need to make it so laughably inefficient to run AI ground-side that no one bothers.
Extremely high temperature semiconductor research to run datacenters in dyson swarms around stars/black holes is fundamental research for human survival at this point.
It seems that the only benchmarks that could be automatically constructed are those for tasks that are themselves verifiable (theorem proving, coding, etc).
Non-verifiable tasks require humans to define what exactly a good job is.
The RSI hypothesis is dependent on an assumption that progress on verifiable tasks generalizes to non-verifiable tasks.
1. Performance engineering ability 2. Ability to reconstruct visuals in code alone 3. Ability to orchestrate cheaper models towards a given task 4. Pretty much any verifiable engineering task
The loop is pretty clear, have AI's work on pre and post-training, then benchmark their outputs against these tasks to determine how to steer AI progress. The non-verifiable stuff isn't on the critical path towards these accoutrements anyway.
It’s a small distinction, but I think it creates a meaningfully less challenging requirement.
But models are really good at being told “here is a paper, check the implementation against the claims in the paper and munge my data to work with it”. On the plus side, many seminal papers have really simple implementations. Like the meat of many CVPR papers is a derivative repo + one function modified. Often they can fix things or backport improvements from more recent work. And they can definitely re-implement things that ought to work, but the authors never published.
On the other side of the spectrum we see people who still want evidence and kick their AI predictions out to a more distant future by pointing at current limitations. This group seems to move the goalposts as needed to support their argument with each new advancement. As the author of the article pointed out, he has moved from this camp to the first one. I have also moved to the first one.
String theory is one that comes to mind.
The behavior of these frontier labs is much more telling than anything else.
All that fuss about the Mathematics problems solved was just LLMs writing Lean code. And besides that, agents are not anything remote to human in terms of behavior and accountability. They are like silly toddlers with a lot of knowledge.
I think AI is not coming for all white collar jobs, it's only coming for jobs that can be solved with code.
One can argue that they will be able to do brain surgery by writing code that execute instructions in a robot arm scalpel. Only time will tell.
Sorry for the rambling, just puking my thoughts.
Absolutely plausible that this math avalanche was 1 cm deep, and harder problems are not solvable without new math theories that have to be constructed with a lot of understanding needed and taste (human heuristic).
Seemingly from you, as a strawman. He doesn't seem to make such a claim in the article.
So he's not an expert. Just a guy off the street. Got it, I'll pass on the podcast then thanks.
“Great minds discuss ideas; average minds discuss events; small minds discuss people.” -- Eleanor Roosevelt
you dont need to delcare this as if anyone cares what podcasts you consume.
If someone has interesting ideas, or a reasoned argument, do you require their credentials before considering what they've written?
It seems likely we end up with an even more extreme tech oligarchy than we already have when superintelligence arrives. And that's only if we manage to solve alignment and it doesn't do something more horrible.
What he says still doesn't map to my daily practice. I use Astra and Fable extensively and they need an incredible amount of guidance and harnessing to produce fairly simple systems. Any decent code is produced with tons of static tooling, agent files, harnessing, and even then, there are myriad mistakes. My main point of skepticism is whether software engineering is verifiable at all, once you factor in architecture, code design, let alone the human element in usability, requirements, etc.
As far as I'm concerned Dwarkesh could be prey of AI psychosis... I don't think a proxy metric like the average daily spend of AI researchers going up proves anything.
The technology has undoubtedly advanced but for me it's still a coin toss on whether we're in an exponential or an s-function.
In my experience, s-curves are much more prevalent in nature than exponentials.
IOW, if an S-curve looks exponential for your entire life, and the life of your children and their children and theirs, does it matter that it's technically an S-curve and not truly exponential? I would argue that no, it doesn't practically matter for the people in that scenario, both options are indistinguishable and change everything completely.
I've been building neural nets in C as a past-time hobby, and I get the feeling that I've been peeking under the curtain too much. The magic is going away. I know they have been getting creative with training, different types of neural nets, and so on. I don't know that I want these organizations working on building an actual sf-like AI, because that might actually human-ending disastrous, but this side road they've gone down isn't going to get there from where I'm standing.
I'm also starting to worry that all I do is regurgitate words and have no real understanding either :)
Lots of companies think they are reaping similar rewards from ai coding and research but it has failed to lift bottomlines or create any new products. My last two employers are gungho on ai and ppl have been producing unfathomable amount of ai output. To a causal observer it would look like we have really accelerated but this has had no visible impact on any of the products that employers produce other than make them shittier.
Whats different about product development at athropic that ai is particular suited for not generating slop.
https://sponsorradar.com/channels/dwarkeshpatel
And that does not include Substack bulk subscriptions.
Isn't it already here? Infallible God-King Trump and all his ultra-genius tech-bro buddies delivered superintellgence already, and due to exponential growth it should already exist by this time, no? By this time next week it will have solved all problems and made humans obsolete.
Seems far simpler than asking it to make super-human intelligence since nobody appears to know how to do that, and tons of people already know how to successfully start companies and make money.
My backup plan is to still be here in a years time, not rich, reading about the latest forecasts about how the singularity is coming real soon now.
https://www.politico.com/news/2026/09/29/trump-signs-executi...
Nothwithstanding genuine one-shotting, mathematics is certainly a great propaganda angle for the AI Bros. Ivory tower image. Not obviously directly applicable to the real world. Theoriticians solving theoretical problems because the problems are there and tenure. That’s the perception, anyway; highly-educated whiteboard writers, not salt of the earth workers. Then you frame that crowd as jealously guarding their theorycrafting fiefdom. But it’s no match for the great AI. And then they go into group therapy next week.
Trump's mistake is using the word to denote AI in general when it already had an established meaning.
Strange how this guy can be CEO of anything. All you hear management types ever blubber about is integrity ... Clearly there's a problem here.