reasonableklout

That's fair, but almost everything so far that models can do much faster and deeper than humans can have been by using the same methods that humans do. For example, with cybersecurity, human researchers could if given enough time find every exploit that allowed the OAI models to escape the sandbox via Artifactory RCE and breach Hugging Face. But still there is a step change in what's possible by doing this at an industrial scale and stringing together all the solutions. The implications of applying existing mech interp capabilities this way could be very significant.

It would likely fall under the same policy that causes latest Opus/Fable to fallback to less capable models when working on ML code

Did Steven unplug the racks that were hacking Hugging Face and taking over OpenAI eval clusters in the multi-week window where it was happening?

I think the idea is that this leads to a much faster, non-auditable form of recursive self-improvement where each generation of models can directly program the weights of the next generation (as opposed to writing and running training code).

We already don't know how to ensure models always do what we want them to do, and moreover society still hasn't had time to absorb the capabilities from the current generation of models, so speeding things up and making things less transparent seems like it'll pretty clearly increase risks from AI loss of control, societal impacts of automation, etc.

It might also be a side effect of training models to help out with mechanistic interpretability, which I guess would help out a lot with recursive self-improvement.

What are people misunderstanding?

This is the first time it dawned on me that mech interp is a dual-use technology which can be used to improve capabilities. Do you think overall this development will improve human understanding of AI models, or will it lead to faster capability advancement using weight-space changes that no humans read?

It is also a much less interpretable and auditable path towards recursive self-improvement. At least with training code, human engineers can inspect and reason about the output.

Well, this isn't an argument or an essay in the typical LessWrong style, it's a short story. I thought it was really good, finished it all in one sitting. And I also usually don't get even halfway through most LW posts.

What are you planning to run on it?

What are we even doing here?

Gift link: https://www.nytimes.com/interactive/2026/10/28/style/gen-z-j...

I suspect the author would respond with some form of "displaced people without capital will take on roles in the relational economy, or as human liability for the robots." I agree the material leverage of such jobs seems radically diminished.

Or we can show some compassion, and work towards a society that both benefits from technology and doesn't discard people at a moment's notice.

I think the argument is that OpenAI should have done the work of cleaning up the proofs in the first place before dumping them onto the math community (or worked with mathematicians to guide the research process). It's as if an energy company dumped 20 tons of waste next to a village and the townspeople decided to clean it up. "How does this benefit the energy company" might not be the right question to be asking.

Interesting, here Sebastien Bubeck comes across as a well-meaning if ultimately powerless figurehead for the OpenAI corporate machine. You can see token concessions to the mathematical community (like the withdrawal of sponsorship from the Mathathon) that someone on the inside had to advocate for but in the end it's all swamped by events like the October 6 proof dump. And people like Bubeck come up with their own rationalizations for staying, even when they are fully aware of the effects.

> BUBECK: 100% honest answer: personally I would like to spend my time on the other areas I care about, which are probably not your cup of tea, but like, healthcare. if we have a good machine, accelerating medicine seems like a good idea.

> NIHIL: And your corporation would botch it up exactly the way it’s botched up math academia. We know this.

> BUBECK: :-(

> NIHIL: You have left a trail of destruction at your wake. I mean I don’t blame you. But that’s what it is.

It harms people in any area that values authenticity because it creates a market for lemons. People will get halfway through a substack post before they realize there is no human behind the words. Again, this is why apps like Pangram are becoming popular. The end result is that people end up paying a tax to find the same value as before.

"Strictly a positive" is also a strange choice of words when the technology is ripping out the livelihoods of so many creators. I would accept "it has tradeoffs, but is a net positive over the long run". But the former is objectively false.

Yup. The headline is also about opposing more datacenters. The AI datacenter boom is now approaching 6% of US GDP and it's not surprising so many are feeling unnerved about the massive footprint of this tech on not only the economy but their physical spaces.

I don't think you understand the value that people actually look for in creative media.

First, people seek authenticity. In written content, people want to know there is a real human author thinking about every word, and in photos, people do not want to be deceived by deepfakes. It's much harder to tell in a world of AI slop, hence the rise of apps like Pangram. This is a significant negative externality.

Second, people who invest their time learning creative fields seek recognition for all the hard work they have put in. Now their skills are devalued, again because of slop. This is another negative externality.

Having more pictures and videos is nice in some ways but your head is in the sand if you think it's all roses.

But in many cases people are being forced into it. If you're a student in a college class of 20 where 1 person is using ChatGPT to cheat, you better bet you're going to start cheating too, unless you want to end up lower on the grading curve.

But why would these same trends not also vastly improve the capabilities of the top hardwares and ability of the trillion-dollar companies to leverage their economies of scale?

I'm all for the idea that AI capabilities can be democratized and help ordinary scientists outside of a handful of labs. But it seems there would need to be some checks against Big Tech and it won't happen by just letting the free market do its thing.

From https://x.com/astuyve/status/2113338192460238954:

A high severity vulnerability in "the most widely deployed DNS software on the internet" just dropped

Nothing for you to update so I guess unless you own a public DNS you can go back to patching those kernels

Even generation of plausible human slop took time and required skills correlated with domain understanding. So the scale is not at all the same.

Zvi is a rationalist blogger who writes a lot of takes on AI news, here's recent coverage of how Washington is reacting to AI x-risk going mainstream with Jacob Coxon, the Hugging Face incident, etc: https://thezvi.substack.com/p/the-ai-preference-cascade-reac...

Because it's an interesting case study of how quickly individual top human experts can understand and verify these mostly-autonomous breakthroughs in math and science. In this case, the world expert on integer multiplication algorithms has taken two weeks and only understood a fraction of the generated paper.

Does the model get things wrong? Or does it just give up in trying to completely understand some very messy codebases?

> Every insular community develops its own self-contained logic, its assumptions that fit tightly together but may miss the broader picture. At Anthropic, this is compounded by the long hours that everyone works, the secrecy associated with its unfathomably precious intellectual property, and the fact it really does have inside information, even if it’s just the next edition of Claude.

Reminds me of Daniel Ellsberg's advice to Henry Kissinger about how learning information kept secret from the public and even much of the government can affect your mind: https://www.motherjones.com/kevin-drum/2010/02/daniel-ellsbe...

> “In the meantime it will have become very hard for you to _learn_ from anybody who doesn’t have these clearances. Because you’ll be thinking as you listen to them: ‘What would this man be telling me if he knew what I know? Would he be giving me the same advice, or would it totally change his predictions and recommendations?’ And _that_ mental exercise is so torturous that after a while you give it up and just stop listening. I’ve seen this with my superiors, my colleagues….and with myself.

Other countries also have a concept of the corporate death penalty. But it has not happened in the US for something like a century.

The context for this article is Jacob Coxon's appearance on The Daily Show where he tries to describe the utopia that AI labs are aiming for to a skeptical Jon Stewart.

Discussion on X: https://x.com/ezraklein/status/2112584886695481831

Also covered in https://techcrunch.com/2025/03/05/eric-schmidt-argues-agains...:

> “[A] Manhattan Project [for AGI] assumes that rivals will acquiesce to an enduring imbalance or omnicide rather than move to prevent it,” the co-authors write. “What begins as a push for a superweapon and global control risks prompting hostile countermeasures and escalating tensions, thereby undermining the very stability the strategy purports to secure.”

> ...

> Schmidt et al. introduce a concept they call Mutual Assured AI Malfunction (MAIM), in which governments could proactively disable threatening AI projects rather than waiting for adversaries to weaponize AGI.

> Schmidt, Wang, and Hendrycks propose that the U.S. shift its focus from “winning the race to superintelligence” to developing methods that deter other countries from creating superintelligent AI. The co-authors argue the government should “expand [its] arsenal of cyberattacks to disable threatening AI projects” controlled by other nations as well as limit adversaries’ access to advanced AI chips and open source models.