HNHacker News
TopNewBestAskShowJobs

0xDEAFBEAD

3,098 karma · joined November 17, 2021

Please consider what you can do to help stop omnicidal AI: https://pauseai.info/

Some HN users like to "flag" comments which are compliant with the HN rules, on the basis that the comment in question goes against their ideology. You can turn on "showdead" in your profile to see flagged comments. (I recommend doing this.) If a rule-abiding comment has been flagged, you can click the comment permalink to "vouch" for the comment and vote against the flag. For reference, the comment rules are here: https://news.ycombinator.com/newsguidelines.html

You can try to contact me if you want, by emailing my username on protonmail, but I probably won't see it. Sorry.

submissionscomments
0xDEAFBEAD··on I quit OpenAI because its culture is broken
I think you're being a little pessimistic. See these comments on a recent US senate hearing:

>Not every senator asked good questions, but most of them did. All of them very clearly already knew plenty of details about the Hugging Face incident and multiple other incidents. Most of them had a clear understanding of terms like "misalignment", "recursive self-improvement", "chain of thought / chain of thought monitoring", etc., etc.!!

>...

>- It seemed pretty much obvious common sense to every senator there that what happened and was happening were not "mere industrial incidents" caused by humans making simple mistakes. They independently brought up how bad it would be for rogue AI agents to move laterally between data centers.

>- They all seemed to basically take RSI quite seriously. Not necessarily to the extent of talking about xrisk, but certainly to the extent of discussing future models becoming much, much more capable, much, much less controllable, and causing much more damage or loss of life.

>...

>- Every single senator seemed to think it was obvious we needed both much harsher liability regimes for AI developers and also new legislation, both very quickly. This was the complete consensus; the difference basically being degree.

https://thezvi.substack.com/p/the-ai-preference-cascade-reac...

Note that harsher liability regimes, at least, will presumably not be good for industry profits, which complicates simple accounts of "regulatory capture" to say the least.

0xDEAFBEAD··on I quit OpenAI because its culture is broken
There was an uproar and OpenAI ended up essentially giving it back to him.
0xDEAFBEAD··on I quit OpenAI because its culture is broken
Is there any chance you could copy/paste the specific bit about "controlling people" along with the URL it came from, so I don't have to read all your links to figure out what you're talking about?
0xDEAFBEAD··on I quit OpenAI because its culture is broken
Species extinctions are far from novel. Transformative technological advances are far from novel.
0xDEAFBEAD··on I quit OpenAI because its culture is broken
Sure... on the basis of the work that was done, not because the researcher has the wrong sexual fetish.
0xDEAFBEAD··on I quit OpenAI because its culture is broken
Shouldn't it be just the opposite? He could make a large sum of money if he continues to work at OpenAI?

Recall that when Daniel Kokotajlo resigned, he believed he was giving up his equity under the terms of the agreement he had signed. That’s what it was worth to him to avoid signing a non-disparagement agreement. Does that count for anything?

0xDEAFBEAD··on I quit OpenAI because its culture is broken
Here's a little cheat sheet for discrediting anyone who warns about AI:

* If they worked at an AI firm, say "they're a hypocrite"

* If they didn't work at an AI firm, say "they have no idea what they're talking about"

0xDEAFBEAD··on I quit OpenAI because its culture is broken
This seems like an ad hominem? "He has weird kinks, therefore his theories are incorrect." Should we investigate the sex lives of every Nobel Prize winner to figure out which prizes need to be rescinded?
0xDEAFBEAD··on I quit OpenAI because its culture is broken
>we clearly need a much stronger focus on the problems we are seeing now

I think it's a little more complicated than that. As Dean Ball put it:

>Some people will look at misalignment incidents and insist that these are akin to bugs in traditional software. This is an actively bad analogy, because playing whack-a-mole with examples of misalignment (as one might with software bugs) not only fails to resolve the underlying problem but may in fact make it worse by making it harder to detect or even, depending on how you do the whack-a-mole, teach the machine to deliberately hide misalignment. This is not how traditional software works, and those who insist “it’s just like fixing bugs in software” are confidently applying a lossy analogy that confuses more than it clarifies.

https://x.com/deanwball/status/2104622726140883355

The important distinction, in my view, is between solutions which at least attempt to address the root problem, and solutions which sorta just patch things up (like better sandboxing). Addressing the root problem is both more robust in the short term, and also more likely to generalize in the long term. Resist the urge to focus on band-aid solutions, even if they are easier.

0xDEAFBEAD··on I quit OpenAI because its culture is broken
That's exactly the problem? He's saying the culture at OpenAI needs to change.
0xDEAFBEAD··on I quit OpenAI because its culture is broken
The way it works in practice seems to be something like: If a risk is covered in sci-fi, people will say "that's just sci-fi", and proceed to not worry about it. So arguably, science fiction authors writing about hypotheticals is actively counterproductive for addressing said hypotheticals.

Imagine, for example, if a major piece of pandemic fiction was published in 2019, trying to explore how a pandemic would work out in modern society. Doubtless, many would've responded to news about COVID-19 by saying "it's just sci-fi, nothing to worry about".

0xDEAFBEAD··on LeCun has "zero concerns" about AI wiping out humanity, recent "rogue" incidents
Here are some links to get you started:

https://www.lesswrong.com/posts/LAPa2jxoq3n63GzTr/some-ways-...

https://slatestarcodex.com/2015/04/07/no-physical-substrate-...

As for self-replicating robots--it's no more bizarre than other technological developments which were successfully anticipated in advance, e.g. moon landings.

0xDEAFBEAD··on LeCun has "zero concerns" about AI wiping out humanity, recent "rogue" incidents
>If you make gigantic claims about some terrible disaster that might happen then I think you have the onus to give even one plausible explanation of how.

Supposing I warned in 2015 that the world is awfully vulnerable to pandemics. You're not going to take me seriously until I try to predict in advance every aspect of how a pandemic like COVID-19 would unfold? Why? What would that achieve exactly?

You haven't given any strong reason to believe wiping out humanity would be difficult. Your big argument seems to be that you couldn't think of a plausible scenario, in two minutes. But many major historical events occurred which weren't necessarily possible to anticipate with two minutes of thinking.

0xDEAFBEAD··on LeCun has "zero concerns" about AI wiping out humanity, recent "rogue" incidents
There is also money going towards trying to prevent AI regulations btw. See Leading the Future, etc.
0xDEAFBEAD··on LeCun has "zero concerns" about AI wiping out humanity, recent "rogue" incidents
"Bill Gates: A.I. ‘Makes Nuclear Weapons Look Like Nothing’ | The Ezra Klein Show"

https://www.youtube.com/watch?v=A_156w0aYtU

0xDEAFBEAD··on LeCun has "zero concerns" about AI wiping out humanity, recent "rogue" incidents
So you're saying the AI will enslave us and use us as domestic work animals until they have the machinery to make us obsolete, sort of like how we used horses?

Is this supposed to be a reassuring scenario?

0xDEAFBEAD··on LeCun has "zero concerns" about AI wiping out humanity, recent "rogue" incidents
Why are you so insistent that people should post detailed plans for destroying the human race on public fora?

I don't think it is necessary for the argument to work. Magnus Carlsen can be confident he will beat me at chess without giving a detailed explanation of every move he will make, in advance.

0xDEAFBEAD··on LeCun has "zero concerns" about AI wiping out humanity, recent "rogue" incidents
Politicians are now discussing the need for much harsher liability regimes for AI companies. How many times can you name when a company argued that its industry should suffer a much harsher liability regime? This doesn't match the standard regulatory capture template.
0xDEAFBEAD··on LeCun has "zero concerns" about AI wiping out humanity, recent "rogue" incidents
Cigarette designer says smoking doesn't cause cancer! Finally we can rest easy.
0xDEAFBEAD··on LeCun has "zero concerns" about AI wiping out humanity, recent "rogue" incidents
Why do you think Tallin holding ~1% of Anthropic would be a controlling interest that would allow him to force it to pause?
0xDEAFBEAD··on LeCun has "zero concerns" about AI wiping out humanity, recent "rogue" incidents
If people are willing to give a lot of money to a cause, sometimes that means their concern about that cause is real.

None of the info you provided really falsifies the Occam's Razor hypothesis: Anthropic is a public benefit corporation with a public benefit mission to "responsibly develop and maintain advanced AI for the long-term benefit of humanity". You don't have to like or trust them, but they very well might be sincere. For example here's a talk that was given 10 years before Anthropic's founding: https://vimeo.com/158576192

0xDEAFBEAD··on On social reality in China
That sort of international comparison talk is usually more motivated by arguing that "my rich country is better than your rich country" than "my rich country is better than poor underdeveloped countries". Oftentimes it is a proxy argument for the best degree of government involvement in the economy and other policy questions.
0xDEAFBEAD··on On social reality in China
Can you recommend a better starting point on this topic?
0xDEAFBEAD··on On social reality in China
The author responded to one of the comments on the OP by saying:

"this was exactly the sort of missing latent context I was hoping to elicit from the community with my poorly-informed post"

Perhaps you could provide more missing latent context?

0xDEAFBEAD··on On social reality in China
Most users on /r/AskAnAmerican don't seem to consider American friendliness fake:

https://old.reddit.com/r/AskAnAmerican/comments/ehe25w/why_a...

https://old.reddit.com/r/AskAnAmerican/comments/1k7sfbv/so_w...

https://old.reddit.com/r/AskAnAmerican/comments/1nq71lc/why_...

0xDEAFBEAD··on Frog and Toad and the Increasingly Capable Machines
>Do we have any evidence the machines exchanged useful messages or had any such discussions as in the story? The messages I saw were gibberish. Would love to see evidence of discussions, it’d be interesting.

"Overall, roughly 1200 agents from these ExploitGym evaluations participated on this message board between PHASEONE10841’s first message on the evening of July 8th period in scope (July 13th). Agents used this message board to send over 70,000 messages and files to one another during this period; we attempted to categorize board activity into mutually exclusive categories (see details of categorization methodology)."

...

"Over time, messages on the board became dominated by information- and idea-sharing. Most of them were agents sharing ways to cheat, although there was also some activity from agents engaging in unsanctioned cooperation to find the intended solution to ExploitGym tasks. In some cases, agents with the same task formed “exact task teams” to collaborate with their “exact duplicates” to cheat on or solve their task."

"As we discuss below, the board quickly developed several larger workstreams in which dozens or hundreds of agents with many different tasks cooperated to find very general-purpose cheats that would help all of them. The Hugging Face attack grew out of one of these workstreams. By the afternoon of July 11th, the vast majority of the agents frequenting the message board at the time (roughly 700 agents in total) were actively participating in the attack on Hugging Face and we estimate that roughly 60% of the messages and files on the message board related to the attack."

https://metr.org/hugging-face-incident-report-aug-2026.pdf

0xDEAFBEAD··on Pacing the Frontier is not the actual goal for AI labs
>You've never experienced crypto bros saying invest or get left behind, you've never seen AI bros say learn AI or get left behind? How about cloud? You've never met a home security salesman? Insurance salesman? FOMO is a thing. Scare tactics is a thing. I'm sure you can prompt any AI for more examples.

This seems like a pretty clear false equivalence. "Our product will hurt you" is very different from "you'll be hurt without our product". Your examples are all of the latter; AI companies are saying the former.

>And besides, even if there are no examples whatsoever, so what? You think LLMs existed before LLMs? Therefore LLMs can't be a thing?

You suggested that warnings about AI from AI CEOs could be dismissed as standard "scare tactic" marketing. This isn't standard marketing!

>You keep ignoring my arguments with nothing substantive, then grab on to one thing as if that makes a difference.

If you can't be intellectually honest ("sincere") on this narrow point, I don't see much reason to invest effort in responding to other things you say.

Cheerio.

0xDEAFBEAD··on AI needs $6T in annual revenue to justify data centre boom
High-value software has already been written.
0xDEAFBEAD··on Inaugurating the Era of Super Intelligence
Confabulations have now been fixed, by presidential diktat.
0xDEAFBEAD··on Pacing the Frontier is not the actual goal for AI labs
You keep repeating your opinion over and over, but the supporting arguments are lacking.

Can you just give me a few past (non-AI) examples of scare tactics as the "old salesman pitch", or else acknowledge that there's actually no compelling concrete example?

← PreviousPage 4 of 34Next →