nopinsight

"It is extremely sad that this didn't end up as an example of how the labs could cooperate/coordinate, because the stakes will be so much higher in the future." -- Sholto Douglas, an Anthropic researcher [1]

"Strong agree. I know that there is rivalry between the labs but it's important that we learn to work together given what's coming. <quote tweet [1] above>" -- Noam Brown, an OpenAI researcher [2]

We all should heed the implied warnings of these top researchers about what's coming. The world is far from ready and everyone who can should pitch in.

[1] https://x.com/_sholtodouglas/status/2097224624274911368 [2] https://x.com/polynoamial/status/2097225279366414541

I suspect an AI, possibly a successor to current LLMs, will achieve that by the early 2030s. It might help illuminate many mysteries in math and beyond for us all.

As someone who's always loved synthesizing ideas and having lightbulb moments, I find the headline flattering. I wonder, though, if there's any more rigorous and general analysis (hah!) of the complexity of these two modes of thought.

EDIT: Fable 5 turned up some relevant references:

[1] Richardson, D. (1968). "Some Undecidable Problems Involving Elementary Functions of a Real Variable." Journal of Symbolic Logic, 33(4). — Differentiation is a simple recursive algorithm; deciding integrability in elementary terms is undecidable in general.

[2] Risch, R. H. (1969). "The Problem of Integration in Finite Terms." Transactions of the American Mathematical Society, 139. — The partial recovery: a (semi-)decision procedure for a restricted class.

[3] Guilford, J. P. (1967). The Nature of Human Intelligence. McGraw-Hill. — The divergent vs. convergent thinking distinction, the classic psychometric cousin of synthesis vs. analysis.

[4] Anderson, L. W., & Krathwohl, D. R. (Eds.) (2001). A Taxonomy for Learning, Teaching, and Assessing (revision of Bloom's taxonomy). Longman. — Moved "Create" (synthesis) to the top of the cognitive hierarchy, above "Analyze."

[5] Aaronson, S. (2011). "Why Philosophers Should Care About Computational Complexity." arXiv:1108.1791. — Argues complexity asymmetries (verification vs. generation, P vs. NP) bear directly on questions about cognition.

Someone also input lots of pre-GPT texts to Pangram and they were all clear. The point is Kramnik’s claims are much weaker and not comparable.

To my knowledge, Pangram has a very low false positive rate approaching zero. (False negative is another matter.) I’m not sure that’s what you want to use as the analogy here? (I don’t know much about this Kramnik situation.)

Quite frustrating to see all these cynical, borderline-irrational comments on HN. Maybe I should do what pg and other ex-HN contributors have done--avoid taking part in the discussions here.

The level of discourse here has dropped so much.

At least I should stop replying to people hiding under throwaway accounts.

Demis' bar is high and he stated clearly multiple times: AGI should be capable of inventing truly novel things. Examples he gave included the Theory of General Relativity and the game of Go.

Just a taste of what's to come: https://openai.com/index/model-disproves-discrete-geometry-c...

Anthropic's latest model also solved this 80-year-old problem that eluded many expert mathematicians, in a different way, according to one of its employees.

Remember we still have 4 years until 2030.

No one rational ever said software engineers were going to be replaced in 6 months. Some people said AI will automate 90% of coding in 6 months and they were not far off (and accurate in some contexts, e.g. startups).

Ten years ago, most of the skeptics in these threads would have said the success of AlphaFold 3 was impossible.

"Ten years from now, I think we will realize that we were standing in the foothills of the singularity now... I believe that we're only a few years away from [AGI], maybe 2030 plus or minus a year...

I think [AGI] will be an enormous transformative technology, it's going to effectively be a new human era...

We can feel this year, I would say, even though I've been working towards this for 30 years, I think this year with the way the agents are working and tool use, it started to become really useful, still early days of it, but genuinely useful in people's workflows...

And it's not any one thing, it's several different technologies, several use cases, several things that I thought were maybe a bit further out, turned out to be now, that are coming together that make me feel that in aggregate.

I think society needs to hear that because we don't have long to prepare for what that means." -- Demi Hassabis, CEO of Google DeepMind & Nobel laureate

Source with interview clip: https://x.com/deredleritt3r/status/2062223035940139253

EDIT: Ten years ago, most of the skeptics in these threads would have said the success of AlphaFold 3 was impossible.

"Ten years from now, I think we will realize that we were standing in the foothills of the singularity now... I believe that we're only a few years away from [AGI], maybe 2030 plus or minus a year...

I think [AGI] will be an enormous transformative technology, it's going to effectively be a new human era...

We can feel this year, I would say, even though I've been working towards this for 30 years, I think this year with the way the agents are working and tool use, it started to become really useful, still early days of it, but genuinely useful in people's workflows...

And it's not any one thing, it's several different technologies, several use cases, several things that I thought were maybe a bit further out, turned out to be now, that are coming together that make me feel that in aggregate.

I think society needs to hear that because we don't have long to prepare for what that means." -- Demi Hassabis, CEO of Google DeepMind & Nobel laureate

https://x.com/deredleritt3r/status/2062223035940139253

With incessant advances in robotics, how long would that continue to be the case?

Should we start preparing for something that could be world-changing in the next 10-20 years?

Things in the real world often take longer than expected. Still, in cities where Waymo operates, many people routinely ride autonomous vehicles and prefer them.

For software, however, a rapid turn is often a possibility. See: AI for coding over the last 3-4 years.

AI autocomplete --> AI coding assistants --> vibe coding --> agent orchestration

Coders can now accomplish work that used to take a week or longer in a couple of hours, with the right tools and skills.

---

A key issue the article implies is that the real world increasingly runs on software.

I have watched Dario’s interview at WEF referred to in the article and I am quite certain Dario didn’t say that. He talked about AI automating most coding already or soon, not software engineering as a whole.

He did say a few months later in an interview in India that AI will eventually take over most of SWE tasks.

—-

My statement on startups is largely about automating coding by SWEs. My startup also uses AI to automate part of technical specifications and code review but I am not sure how widespread that is.

This is already the case for many startups. In fact, the figure might be closer to 100%. The work shifts to requirements analysis, high-level specifications, and final review instead (after AI code review).

I assume you're using the "regular" Pro version of Gemini 3.1 for the above, rather than the Deep Think mode, which is more comparable to GPT-5.5 Pro. To my knowledge, regular 3.1 Pro is a tier below and often makes mistakes.

Moreover, there's no reason to believe the progress of LLMs, which couldn't reliably solve high-school math problems just 3–4 years ago, will stop anytime soon.

You might want to track the progress of these models on the CritPt benchmark, which is built on *unpublished, research-level* physics problems:

https://critpt.com/

Frontier models are still nowhere near solving it, but progress has been rapid.

* o3 (high) <1.5 years ago was at 1.4%

* GPT 5.4 (xhigh), 23.4%

* GPT-5.5 (xhigh), 27.1%

* GPT-5.5 Pro (xhigh) 30.6%.

https://artificialanalysis.ai/evaluations/critpt.

There are reasons, based on machine learning related theory, that justify the belief.

Economics will likely sort itself out through optimizations, which are also highly plausible.

Given the capabilities of upcoming LLMs, I suspect that by mid-2027, most competent companies, outside specific niches, will not hire and might fire any non-senior “generative AI vegetarian” software developer.

Note: I agree with others that another term should be used instead of ‘vegetarian’. “LLM vegetarians” do not hold the same moral values as vegetarians.

From the author’s earlier essay:

“A good way to describe myself is as a generative AI vegetarian. You can find a fuller explanation—and many, many links—at the above essay by Sean Boots, which I agree with almost 100%.”

—-

Given the capabilities of upcoming LLMs, I suspect that by mid-2027, most competent companies, outside specific niches, will not hire and might fire any non-senior “generative AI vegetarian” software developer.

> "a broken clock is right twice a day."

The combinatorial nature of trying things randomly means that it would take millennia or longer for light-speed monkeys typing at a keyboard, or GPUs, to solve such a problem without direction.

By now, people should stop dismissing RL-trained reasoning LLMs as stupid, aimless text predictors or combiners. They wouldn’t say the same thing about high-achieving, but non-creative, college students who can only solve hard conventional problems.

Yes, current LLMs likely still lack some major aspects of intelligence. They probably wouldn’t be able to come up with general relativity on their own with only training data up to 1905.

Neither did the vast majority of physicists back then.

They probably want to select for high-quality ads without having to be responsible for filtering issues, whether false positive or false negative, which will adversely affect their reputation with consumers and advertisers. They probably wait until they have enough data/experience to do that properly.

Ord's frontier-cost argument is right as far as it goes, but the piece doesn't engage with the counter-trend: inference cost for a fixed capability level has been falling faster than Moore's law. Pushing the frontier will likely keep getting more expensive and concentrated among a few players, while the intelligence needed for more mundane tasks keeps getting cheaper.

That raises a question: if practical-tier inference commoditizes, how does any company justify the ever-larger capex to push the frontier?

OpenAI's pitch is that their business model should "scale with the value intelligence delivers." Concretely, that means moving beyond API fees into licensing and outcome-based pricing in high-value R&D sectors like drug discovery and materials science, where a single breakthrough dwarfs compute cost. That's one possible answer, though it's unclear whether the mechanism will work in practice.

A growing number of non-technical managers are now using Claude Code to build small custom software. A larger share will use Cowork to automate routine business tasks. Claude Cowork will become easier to use and more automated over time, as it learns the user's preferences, just like a good executive assistant does.

Granted, it's possible that a majority of people will not acquire proper 'manager brain' either and we'll see how that pans out. Evolutionarily, managerial skills are much more aligned with what many hunter-gatherers might learn as they mature and become more of an advisor than a doer.

Even if only 10-20% of people end up using multiple autonomous agents regularly for their work and business, that will change the economy. Contrast this with <1% of people who develop software professionally.

I don't think 'software brain' is required for non-coding tasks. Rather, it requires 'manager brain', the ability to delegate, direct, and review the output. Manager brain is more prevalent than software brain and likely learnable by many knowledge workers who don't yet have it.

> LLMs are better at 'fuzzy' things like writing specs or documentation than they are at writing code.

At least for writing specs, this is clearly not true. I am a startup founder/engineer who has written a lot of code, but I've written less and less code over the last couple of years and very little now. Even much of the code review can be delegated to frontier models now (if you know which ones to use for which purpose).

I still need to guide the models to write and revise specs a great deal. Current frontier LLMs are great at verifiable things (quite obvious to those who know how they're trained), including finding most bugs. They are still much less competent than expert humans at understanding many 'softer' aspects of business and user requirements.

Jensen Huang said he commands thousands of AGIs but still feels pretty useful.

Founders and CEOs are still needed to set direction, bring unique vision to life, and build relationships for long-term partnerships—-as long as humans still control the economy, that is.

> nobody is sure if even their metered pricing is profitable

This is most likely wrong. Lab executives insist that serving tokens is profitable. It's the cost of training next-gen models that requires them to keep raising ever larger rounds. More importantly, many independent providers price tokens of open-weight models at a fraction of Anthropic's prices.

It’s not that simple since each problem is supposed to be distinct and different enough that no single program can solve multiple of them properly. No problem spec is provided as well iiuc so you can’t simply ask an LLM to generate code without doing other things.

LLMs in some form will likely be a key component in the first AGI system we (help) build. We might still lack something essential. However, people who keep doubting AGI is even possible should learn more about The Church-Turing Thesis.

https://plato.stanford.edu/entries/church-turing/

You are assuming that superintelligent AI will follow every command of users outside the labs.