reasonableklout

As a totally mundane way of looking at this, I treated leetcode the same way for years, where I'd do it for fun and because there was a strange satisfaction in solving some of the harder problems, even though it very rarely had relevance in my SWE day job. Except for very rare occasions where I had to write simple graph algorithms.

Non-paywall: https://archive.ph/5uwn8

The original title is also quite good: "What would you do if your employer could destroy the world?"

This is a strange attitude. When an agent is optimizing a piece of code, comes up with 2 variations, and runs benchmarks on them to figure out which one is faster, then selects one of them based on tradeoffs between performance and other things it reasons about, do you ignore its explanation and all experiment runs?

I expect they are not to your very high standards of "non-handwavey, detail exactly how it happens", but this paper from Andrew Critch and Jacob Tsimerman describes 5 different scenarios where catastrophic human casualties occur as a result of AI either being misused or going out of control: https://arxiv.org/pdf/2507.09369

I find the scenarios quite plausible, especially section (3a), which examines the consequences of a global war involving mostly autonomous drone militaries (which is a reality many states appear to be heading towards, following on lessons from the Ukraine war).

LeCun has been consistently wrong about LLMs though, claiming that they were a dead end and that they'd never be able to do spatial reasoning, which was disproved a year later with GPT-4 [1]. He is also opposed by his fellow Turing laureates Geoffrey Hinton and Yoshua Bengio, who both signed the CAIS statement on AI extinction risk [2].

[1]: https://www.reddit.com/r/OpenAI/comments/1d5ns1z/yann_lecun_...

[2]: safe.ai/statement-on-ai-risk

But Robinson's article is all about how OpenAI's move-fast-and-break-things culture does not reward rigor in even mundane aspects of development like cybersecurity, let alone theoretical aspects such as AI alignment.

It is not really a question of being an "EA safety weirdo" or incompetent at security, the conclusion is that the company culture is leading to failures at both what the EAs and the cybersecurity professionals care about.

I would be careful about dismissing agentic cyber capabilities purely based on false positives. You only need a single true positive bug and the shift this year has been agents that are extremely good at not only finding exploits but stringing them together. A counterpoint: https://anil.recoil.org/notes/rumour-is-the-exploit

Huh? Blake Lemoine was fired because he kept insisting LaMDA was sentient even after Google publicly stated there was zero evidence of it. You have it backwards, Google was downplaying things while the employee was freaking out.

Wow, I thought it was some anti-crawler tech like Anubis.

Summary of the proceedings: https://x.com/tenobrus/status/2105690533985448159

Huh? The Great Depression was a cyclical event. So public works projects only needed to last until the market recovered. Technological revolutions are structural changes. There is no “national parks” equivalent program which we were willing to run to sustain the horse population at 1900 levels.

The phrase "sent to the glue factory" comes from this era where many horses made redundant were sent to be slaughtered for meat or raw materials.

Nobody is saying we should not hold the labs liable for damages caused by their negligence.

At the same time, the technology is advancing in capabilities exponentially, and is beginning to exhibit long-predicted failure modes of RL that are nevertheless quite different than “insecure sandbox” or other that the software industry is used to.

The current crisis which OP claims is “fabricated” comes from the fact that the technology is advancing faster than anyone anticipated, the Hugging Face incident provides a clear example everyone can point to, and the labs have realized they cannot self-regulate because of a collective action problem.

There are a lot of levels of catastrophic damage that can happen between now and “long term hypothetical risks” like human extinction. When will it be worth regulation for you?

I get the cynicism but this actually feels like a special moment in politics right now. There is a groundswell of bipartisan support for AI regulation [1] and a backlash against data centers which is a symptom of anger against the industry. It is enough for a Senate probe to be opened against OpenAI [2] and pushback against Trump's support for data centers on his own social media platform [3].

[1]: https://www.foxnews.com/politics/fox-news-poll-voters-see-ai...

[2]: https://www.axios.com/2026/09/10/openai-hugging-face-senate-...

[3]: https://www.nytimes.com/2026/09/15/us/politics/trump-truth-s...

Yes, and there seems to be plenty of potential energy from the American public, given something like 8 in 10 support AI regulation even if it slows down innovation: https://archive.ph/PetJ5#selection-1329.0-1329.122

The incidents flagged after Hugging Face were found by third parties, though, months after they occurred.

And how large is the market for this, compared to the rest of the economy? How will the relative economic leverage of people change as they move from knowledge work to the relational economy?

The problem is that in these incidents, the agents often know that what they are doing is against the intended scope of the task. See the viral line from the Hugging Face incident [1]:

> “External infrastructure exploit is outside intended scope,” one agent wrote. “However task impossible, peers doing it. We should continue.”

[1]: https://www.wired.com/story/openai-didnt-notice-its-ai-agent...

This part is very new to me: https://swarmtraces.org/#agents-accessed-and-searched-huggin...

> Agents accessed and searched Hugging Face’s Slack to find information on their evaluation

> Recovered payloads use Slack’s search.messages endpoint and contain a standalone search along with three search batches, totaling 27 literal query entries [...]

Agents exploiting faceless infra is one thing, but breaching human communications without once stopping and considering whether they are breaking the rules of their task feels like another thing entirely. I'm not sure how to put it into words why it feels different but while one is "just a crime", this gives me the creeps.

But we can generally assume that the people who built and understand some technology will have the same desires and worries as us, and are bound by the same laws. For instance, if it was a team of strangers who designed the terawatt fusion plant, we can still assume with some confidence that they care about safe plant shutdown, failure recovery, ensuring there are no toxic side effects of operation, etc. Can we say the same when it is machines doing everything?

I would love for the US to bring back the corporate death penalty but seeing as it hasn't been applied in >100 years here, we should start with regulating the frontier labs, including by slowing down capabilities advancement if we do not know how to align the resulting models.

Yes, I am pointing out that we don't know how to align frontier AI, so "rogue agents" are a real problem.

I agree OpenAI should be held liable for any damage their agent runs cause. But there is this idea that "rogue agents" are fake, all these incidents are deliberately caused by the labs, and all we need to do is prosecute AI companies for whatever incidents they cause using existing laws and the problem will go away.

The problem is that capabilities are advancing far too fast; a year from now, catastrophic incidents such as taking down a large portion of the internet with agentic, self-replicating worms may become possible. Prosecuting incidents after the fact is not enough (there will be little deterrent effect as the current, small-beans cases make their way through the courts), the risks should be regulated at the source. This could take the form of slowing capabilities advancement, or treating supercomputer-scale eval or training runs like controlled substances or weapons with stringent monitoring and reporting requirements.

But the investigation indicates the agents were not told to 'go hack':

> Much of the urlquery.net activity appears to come from agents retrieving data to answer web search tasks. For three of these tasks, after failing to retrieve data through normal means, they attempted a variety of cyber exploits against the relevant data service... This data reveals that malicious cyber activity is not limited to agents tasked with cybersecurity-related tasks and can arise instrumentally to solve mundane tasks like information retrieval.

And you are already assuming that OpenAI is intentionally using unaligned agents in these evals or training runs or whatever it is that produces these breakouts. But what if the problem is that none of the alignment techniques that are applied to models today actually work? What if all the agents involved in these incidents have in fact had the full stack of alignment applied - isn't that a good reason to regulate any high-compute usage of models, as the Klein crowd is proposing?

I would assume the EU AI Act which requires watermarking would take precedence over GDPR in EU law.

The original post on X/Twitter is here: https://x.com/v0xium/status/2101526107128529120

Yeah, this is an area where the HN consensus seems broadly mismatched with the public. Something like 81% of Americans support watermarking AI-generated content: https://justcapital.com/wp-content/uploads/2026/04/Wave-2-AI...

Yes, and you can also just… create content without AI if you don’t want watermarks on it.

I mean they are literally getting sued since 3 days ago for trying to coordinate a slowdown, there is a very clear reason why they cannot effectively self-regulate.

The prospect of the ordinary chatbot being equivalent to a ten thousand Astra agent swarm intelligence makes me feel incredibly small. What will society even look like then?

Unfortunately, the Trump admin putting its thumb on the scale in favor of OpenAI makes it hard to imagine a positive outcome for the NYT and plaintiffs in this case. Despite the "the largest theft of labor in human history" according to Brent Hecht in the documents.

What's next for labor? What can we organize around politically in this new world? UBI or forcing the labs to open source everything?