The old "might end the human race" trick used by cynical promoters throughout history
If only investors and buyers weren't so vulnerable to the "might kill you and everyone you ever cared about" ploy
8,135 karma · joined November 4, 2007
No one has any plausible plan or halfway-decent plan for how to maintain control over an AI that has become super-humanly capable. Essentially all of the major AI labs are trying to create a super-humanly capable AI, and eventually one of them will probably succeed—which would be very bad (i.e., probably fatal) for humanity. So, clearly the AI labs must be shut down and in fact would have been shut down already if humanity were sufficiently competent. The AI labs must stay shut down until someone comes up with a good plan for controlling (or aligning) AIs, which will probably take at least 3 or 4 decades. We know that because people have been conducting the intellectual search for a plan for more than 2 decades as their full-time job, and those people report that the search is very difficult.
A satisfactory alternative might be to develop a method for determining whether a novel AI design can acquire a dangerous level of capabilities along with some way of ensuring that no lab or group goes ahead with an AI that can. This might be satisfactory if the determination can be made before giving the AI access to people or the internet. But I know of no competent researcher that has ever worked on or made any progress on this problem whereas at least the control problem has received a decent amount of attention from researchers and funding institutions.
The AIs that might prove fatal to humanity will be significantly different in design from the AIs that have been already widely deployed: for one thing, they will constantly learn (like a person does) as opposed to already-deployed AIs in which the vast majority of the AI's learning happens during a training phase that ends before any widespread deployment of the AI. Also, they will be much better than current AIs at working towards a long-term goal. I say this because I don't want to be misunderstood as believing that Google Gemini 2.5 or ChatGPT 5.0 might take over the world: I understand that those AIs are incapable of a such a thing. The worry is the AIs that are still on the drawing board or that will appear on a drawing board 5 or 10 years from now. There is no need to ban Gemini 2.5 and ChatGPT 5. Since some AI researchers pursue AI "progress" for ideological reasons and will tend to persist stubbornly even after AI research is banned, the best time to ban AI frontier research is now, so that these stubborn ideologues whose research will have been driven underground (because of the ban) but are capable of making a little more "progress" on AI will be unlikely to be able to make enough "progress" to end the world.
Again, as soon as anyone comes up with a solid plan for controlling (or aligning) an AI even if the AI turns out to be more capable than us, the ban on frontier AI research can be lifted as long as the majority of AI experts and AI researchers are in agreement that the plan is solid. No one can say how long this search for a solid plan will take, but IMHO it will probably take at least 3 or 4 decades.
More at
https://intelligence.org/the-problem/
If only investors and buyers weren't so vulnerable to the "might kill you and everyone you ever cared about" ploy
And sometimes I think Star Trek did more damage to the world than Das Kapital did.
Although I agree with you that non-white immigration has had the effect of making the electorate more paternalistic, my guess is that most on the Left would be disappointed to learn that.
The pro-immigrant Left must believe that the immigration helps in the fight against oppression somehow or that it helps the immigrants while not harming the US. I think particularly that they reject (as racist) any suggestion that non-white immigration might harm or diminish the US in any substantial or lasting way.
Some of the comment on HN over the years claim that immigration is essential to keep the US from collapsing (although I don't know whether that view has any influence among Washington decisionmakers). If someone believes that US whites got most of their wealth and privileges by extracting labor from non-whites, then it starts to make sense to believe that the US economy is structurally dependent on exploitation and consequently needs a steady stream of non-white immigrants to extract labor from. (Maybe for example he believes that the children of immigrants are less exploitable than their parents because for example going to college in the US insulates a person from most forms of oppression.)
And like others are saying, it is at Battersea because Apple has about 500,000 sq ft of office space at Battersea. It's their UK headquarters.
It is well known for decades that a Mac user spends many times more on software for that Mac than a PC user spends for software, so it is reasonable to conclude that an iPhone user visits more web pages.
If you work at a defense contractor on warplanes, radar systems or nuclear weapons and you are too free in how you teach or write about your professional knowledge, you go to prison, and it has been that way in the US continuously since WWII. Criminal prosecutions are relatively rare because security-clearance background investigations filter out most of the job applicants who would go on to violate the laws.
Safari started as a fork of KHTML, and Chrome started as a fork of (the open-source parts of) Safari.
"The world will regress to a pre-industrial-revolution lifestyle" seems to be a retreat from "civilization will end" (not that I agree with it).
>travel will be even more risky than in the Dark Ages thanks to fallout
The radioactive contamination caused by Chernobyl has persisted for decades. In contrast, the radionuclides created by nuclear explosions are of a very different nature and persist for weeks. 3 weeks after the end of the explosions, there will be no place except ground-zero craters (i.e., very small areas) where the levels of radioactivity remain at deadly levels. (But I definitely agree with you that during those critical 3 weeks, almost all people who decide to travel will end up dying unless they know a lot about nuclear war and have good radiation detectors.)
To repeat my overall point: the worst-case scenario for the outcome of AI research should be compared to an asteroid strike by an extremely large asteroid, not to nuclear war—because nuclear war is too tame in its effects to make a good comparison.
The reason I got involved in this discussion is that some people use the possibility of nuclear war as one reason for us to go forward with AI research as fast as possible. Such people argue that a sufficiently mature AI technology would eliminate the possibility of nuclear war. I think that is a fallacious argument become AI "progress" is the more potent of the two dangers.
But what HN has instead is extensive discussion about whether people who oppose the continuation of AI research are morally degenerate in one way or another (or weird or ridiculous)—in other words, attempts to cancel them (or laugh them out of the room) before the listener hears their arguments.
Also, if you're going to talk about the corrupting or biasing effects of a flow of money (like the OP does) you should at least mention the absolutely huge flow of money going (from investors and users) to the people building AI—or or the $140 million that has been donated to the "Leading the Future" Network (a pro-AI political action committee) or the $3.5 million OpenAI has spent on federal lobbying or the $3.1 million Anthropic has spent.
In other words, it's not Pascal's wager if the very bad outcome is what you naturally expect to happen.
And what I just wrote applies to not just Eliezer and his allies: I don't know a single person or a single published argument that has ever used Pascal's-wager-like argumentation to argue that AI "progress" needs to be stopped. Eliezer is the author of a blog post titled "Pascal's mugging", but that blog post and any follow-up discussion are completely silent on the dangers of AI.
I believe what I wrote is an accurate summary of information in a report published this month by Anthropic. I haven't seen the report, but have seen this next summary written by someone who I trust to summarize accurately: https://thezvi.substack.com/p/the-bad-guy-with-an-ai-named-c...
OpenAI reported similar behavior last year. The assertion that the other (two) leading US models are being distilled in a similar manner is an assumption on my part. Where I wrote, "Claude or another leading US model" I initially had just "Claude", but it felt silly to imply that the same thing is not happening to the other leading US models.
If that poisonous piece of propaganda had never been published, then today everyone would probably perceive AI correctly as a benign and beneficial.
The Terminator didn't help either, but Cameron probably got the idea of robots hostile to humanity from Rossum's Universal Robots. How else would he have arrived at such an absurd conclusion?
But according to you, the negative opinion is the result "not thousands of individual perspectives but a chorus of connected/aligned people who have all read the same things and talked to the same people and are rewarded implicitly for thinking a similar way" whereas the 1% or 2% of the US population that comprises Silicon Valley are not being rewarding for believing anything particular about AI, don't communicate with each other and form their beliefs about AI through a wholesome natural organic process. Do I have that right?
No. The Chinese labs pay much less to train their models because they distill Western models. Western labs can't do it that way because they'd immediately be sued.
To distill a model requires high-quality prompts (and responses). Here's how the Chinese labs get these prompts: when a user sends a prompt to www.kimi.ai, Kimi immediately forwards it to Claude or another leading US model (using a vast network of laundered subscriptions), waits for the response from Claude, then forwards that response to the user via www.kimi.ai. All the Chinese labs do this. Of course, they never inform the user that their conversation is being routed to a US model.
Beijing is angry at them because they did this indiscriminately without filtering out the conversations of high Chinese government officials and military officers, so now the US labs have those conversations, which contain information useful to US intelligence because the officials and officers assumed the conversations would remain inside China's borders and that the Chinese labs would care about confidentiality.
The Chinese labs release model weights under permissive licenses to get any attention and usage share at all for the model.
It might be that the models have been improving in this sense faster than a human child improves.
What would be their motivation?