reasonableklout

A major update for Gowers, who says until recently he had a psychological aversion towards taking xrisk seriously, as he already had climate change causing him enough anxiety:

> Until recently, I have always found that my anxiety of choice was climate change rather than x-risk, but I was aware that there were a lot of very intelligent people who disagreed with me, and also aware that I had thought much less about x-risk than about climate change. So when I agreed to be on the scientific advisory panel, I also decided that that was the moment that I should think a bit harder about x-risk.

> ...

> I find it very disheartening to read a climate denier trotting out a layer-1 argument when its layer-2 rejoinder has been established for a long time and can be found after two seconds on Google (or, nowadays, an LLM). But until recently, my arguments against x-risk were basically layer-1 arguments, the sorts of arguments that would provoke eye rolls among the people who have thought properly about it.

His twitter TL;DR: https://x.com/wtgowers/status/2111045962911097188

> I don't have a p(doom), but in particular I can't confidently say that my p(doom) is close to zero, which is a worrying position to be in. I also try to encourage mathematicians to think harder about the question and maybe consider working on AI safety.

How is Jensen more reasonable? On the contrary, he absurdly claims that all of Hinton’s predictions have been wrong (Hinton made a single famously wrong prediction about radiologists, but many correct predictions about how neural networks would be able to solve many different types of tasks). On the same Ezra Klein podcast he also claims that LLMs are “just software” and “software breaks out of sandboxes all the time.”

It’s only been a few months since LLMs started solving significant open problems in math. I wouldn’t jump to the conclusion that they have it better than software engineers.

In many ways it will likely be the opposite. Tech firms still need engineers to talk to customers and run experiments. What will the mathematicians be needed for?

It seems far more likely to me that current LLMs have learned generic reasoning capabilities which will allow them to solve effectively infinite problems, rather than whatever vague limit on capabilities you seem to think there are.

It is not the same because the act of problem solving for discovery is now to be done exclusively by machines, as humans are no longer remotely competitive.

If you read the linked article, the author goes into more detail about how doing mathematics also has been an act of communication with past and present human mathematicians. This is very much true when math and science are done by millions of human researchers, but not so when it is done by massive swarms of agents at a couple labs.

I am sure science and math will benefit in the sense of solving more problems than ever before. But you are not really understanding the agony of the humans caught in the change.

Did you read any of the explanations from mathematicians over the last month? What part of them do you disagree with?

For example [1]:

> In many fields and activities, years of training have traditionally served not only to produce a final answer or product, but also to develop understanding and the ability to formulate new questions and ideas. However, building on a vast body of previous human work, AI systems are becoming increasingly capable of producing the results of such work directly, and these goals cease to align.

Or [2]:

> Use of AI tools, when done responsibly, can support not just the initial generation of a solution, but all of the valuable followup activities that traditionally flow out of that solution. But at the current time, the opposite is often occurring: problems are being solved autonomously by AI prompters who have no interest in the broader field itself once their initial target is "solved", and do not understand the AI output well enough to answer questions on the result, give talks, or otherwise interact with the rest of the field.

> ...

> solutions to open problems are now being harvested at large scale in an unsustainable fashion, leaving entire fields of mathematics much less fertile than when such problems were solved in the traditional "Math 1.0" fashion.

[1]: https://mathandai.org/

[2]: https://mathstodon.xyz/@tao/117395268130901862

This is only an improvement if Gemini has made progress on alignment science. Otherwise, we end up with 2 sources of swarms on the loose instead of 1.

I think ridesharing was a net good for ordinary people even if Uber is/was unethical.

However, the counterargument falls entirely flat. "Whatever benefits AI has to offer on a societal level are going to take a long long time to show up". We don't have the luxury of time because AI disrupting communities and livelihoods faster than perhaps any other technology we've ever seen, with the most capital that has ever been put into any tech since the railroads.

Even if you're optimistic we'll eventually get the cancer cures and fusion reactors, doesn't it make sense from the perspective of an ordinary American to regulate the harms like the homework cheating, parasocial relationships, job losses, cyber attacks, etc., especially if it will take years to get the benefits?

This only works if there remains a viable and attractive consumer market for these advanced goods. AI is both an imminent wealth shock and perhaps the largest structural change to the economy in history as it threatens to drive the value of human labor to zero over the next decades.

In American history, when previous shocks like this occurred (e.g. with the Great Depression), we needed massive economic programs like the New Deal to restore consumer demand. When such programs are not forthcoming while wealth inequalities keep increasing, one can look to other countries for what ends up happening; either the disenfranchisement of ordinary people over time or unrest and redistribution by force.

What's interesting to me is that some of these communities are remaining strong despite the AI tidal wave. Although it might be my bubble. For instance, there is still a thriving art scene on instagram which has successfully demarcated itself from AI art. There is also a writing community on substack. Are there hobbyist programming communities out there that can keep the practice of human coding alive?

For the last 2 years the frontier labs have been taking on funding at valuations that can only be justified if they replace vast swathes of human labor (and they have been pretty successful so far at automating large parts of software jobs, if not the jobs themselves yet), so the career doom and gloom is justified.

I think the key question we are facing is that while we cannot control those around us - who often need to adopt AI as an economic necessity - how can we ensure we are still able to find and ensure intellectual communities thrive?

I think for me I have made a conscious decision to continue writing even though lots of writers now use AI. I also find myself gravitating towards platforms like substack which already have a network of people who value human artifacts.

Title is fairly clickbaity:

> Between the lines: The planning involves red-teaming for worst-case scenarios. It focuses mainly on racing to educate members of Congress.

This part is a bit less clickbaity, because it suggests a belief in inevitability that the public does not necessarily share:

> Threat level: While scenario planning is common within many companies and national security settings — the Pentagon has been running war games for decades — the difference is that many top AI researchers and executives believe a major incident is inevitable.

Great point, whole harnesses and concepts like "loops" are obsolete in a matter of months. I think it was just 6 months ago that the GPT-5.x models needed a harness with an edit tool that took arguments in the form of a diff, otherwise the model would have trouble editing any files. Flash forward to 3 months ago and the Claude models now skip the edit tool and use the bash tool with some `sed` command 90% of the time.

It's sort of like a "bitter lesson" for all of software. Attempting to polish our tools isn't very useful when the tools are themselves these fluid minds that get general upgrades every month, and now every couple weeks.

It's a fantasy but I wonder if you were to wave a magic wand and pause model releases for 10 years if some of these areas would develop their own depth.

The author's contention is that the AI-in-software discourse is much less intellectual than software discourse previously was:

> The discourse around software engineering has gotten dumber. It’s like everyone in the industry lost 30 IQ points. People used to talk about compilers, type systems, logic. Now they talk about “prompts”, “harnesses”, “loops”. The discourse is narrower, shallower, and more repetitive. There’s only so many times I can hear about “agentic harnesses” before I lose my mind.

I think the author would probably agree that the profession of software engineering still has important responsibilities in the age of agents. He says so himself about his new day-to-day role:

> In my professional life, I’m happy to move one level up to become a manager of AI agents. I’m literate, I’m a good technical writer, I can describe what I want and let AI agents write the code.

Regarding continuing to practice software engineering, it is a noble ideal, but do you have an answer to the erosion of community described in the article? For instance, do you have suggestions for the author for how they might still feel motivated to invent a new programming language?

I think there are things we can do to preserve our intellectual practice, like intentionally forming communities that only do traditional coding (much like how human-only art communities still exist), but it's useful to suggest these concretely if you have any in mind.

Well, the post is arguing that even if you're free to find another community that fulfills an intellectual need for you, intellectual communities in general are growing weaker or being driven underground because AI is poisoning the commons and removing a source of external motivation.

For example, I enjoy blogging, and want to use the time saved from my software job to blog more. But I'm less motivated to blog these days because there's so much AI-generated slop blog content, and less readers as a result.

Part of this is because there's so much output that readers can no longer afford to spend as much attention on each piece of content. Another part is that it's harder to find fellow bloggers who truly care about the craft. When output was hard, anyone who wrote a lot clearly cared a lot. Now we have to use other signals to find kindred souls on the internet. One way is through social connections and private discords. Another way is through new technology which we can use to reclaim the commons, for example via Pangram which can demarcate what content is human-written vs. AI-written.

The previous OpenAI board already tried. It didn’t work, there are significant internal politics at play here. And in any case, the current 2 top players hate and mistrust each other and do not want the other to win.

The second page of the letter describes in detail how they followed company protocol and took only authorized actions in their communications with METR (which was Tomek’s assigned role as liaison). The OpenAI response on twitter is very vague and just says “there was more to it than the stuff in the letter”: https://x.com/OpenAINewsroom/status/2108441580806025712

We’ll see if OAI’s side ever comes out, but right now there’s nothing contradicting the idea they were fired on a pretext.

This post in the blog comments is quite compelling. Some observations that publishing of research is drying up because results can be easily retrieved at any time. Over the long-term, I wonder whether this will result in accumulation of knowledge grinding to a halt:

> With the latest ChatGPT models, these problems are more equivalent to homework questions: the answer is `in the back of the book.’ I am not discovering new solutions. Instead, I am working on problems whose answer exists and is simply waiting to be retrieved by a user of the model. In fact, I mentioned a problem that I was interested in working on to my advisor and he informed me that he and a collaborator had completely resolved it using ChatGPT – they have no plans to write up the result, so it will sit there until another `researcher’ pulls the proof slot machine.

There is another good point that the most tenured researchers have a sense of what problems are most worth exploring and therefore are more likely to feel excitement than younger researchers:

> I have also heard the contention that math research has `gotten more exciting,’ mainly from established researchers. They have decades of open problems that they care deeply about and want to see resolved. I have no such problems.

The economy can show us whether the conclusion holds, but it does not tell us anything about the premise. Do you believe the point of humans is to produce things for the economy?

This is a bleak view.

One can also say: The point of humans is to produce things for the economy. Now that humans are obsolete for producing things, humans are unnecessary.

Would you agree?

Even in the industrial revolution there were vast swathes of work that were untouched by machines.

It is good to maintain some optimism that people will figure out what to do but there is a difference from rushing towards such a future and blindly assuming it will be good by default. Are there things you think we need to figure out to avoid a bad outcome?

> Interestingly I'd note that the two outcomes you listed (power concentration and loss of control) are dimensionally opposites!

One can imagine a world where both happen. For example, an arms race between two leading labs causes them to accumulate a majority of the world's compute and market share for intelligence, with the race dynamics at some point resulting in a catastrophic incident of a self-replicating botnet taking over a large part of the internet due to a rushed, reckless Hugging-Face style RL run.

If the argument is that we have discovered 200 years of math in 1 year that we might need to throw out for fear that blindly applying its incomprehensible results will lead us to ruin, then I would say: why not settle for 199 years of math in 1 year which can be verified by our human intellectual reserve that we will train on the remaining 1 year?

I think I would feel a lot better if it wasn't a $15 million cannon being fired from a silicon tower inside of the labs across vast swathes of fertile ground that would otherwise be used to train budding mathematicians. For instance, if it was the budding mathematicians themselves who were making these discoveries using their own tools.

I feel something about human nature makes us treat joy of discovery, status, etc. as a source of energy and motivation. I hope we'll find other ways to keep some "strategic intellectual reserve" of mathematicians alive.

Mathematics has it slightly worse, since math is a "closed system" that does not require running experiments, talking to customers, etc.

That said the software folks can teach the math folks a thing or two when it comes to dealing with grief I am sure.

Can you share what makes you so optimistic?

The maths result is cool on one hand (discovering truths of the universe faster), but on the other there are so many bad outcomes that seem likely, from power concentration to loss of control.

I think this topic deserves some empathy.

It is not about AI the technology, plenty of mathematicians are happy to use AI, it is about the AI companies. The tech only exists because of centuries of mathematical tradition in open science. Moreover, the livelihoods of mathematicians depend on research results and ideas which can be developed only by doing the hard work of exploring open problems.

The recent statement from mathematicians is about how the labs have spent tens of millions of dollars (resources even entire math groups at universities can only dream of) on what was essentially marketing bragging rights for their models. This directly harms the math community by depriving them of opportunities for both funding and fertile ground for new ideas, while at the same time being built on top of their entire body of work.

Now, have OpenAI decided to change their practices and stop publishing math just because they are disrupting an entire field? Not really, since this release still dumps a huge number of results to open problems without waiting for human understanding to catch up. But at least they have committed to funding programs and talking to members of the field to work out how to best evolve it in this new world. And we will still get the benefits of results that happen to have applications.

I don't! But there is no such qualifier in the announcement. Perhaps OpenAI ought to filter for the problems that have immediate applications, and leave some that are unclear for people.

Now I'm curious if there is such a site or article that ranks open problems based on votes from human mathematicians.