Statement on AI Risk
safe.ai
safe.ai
If this was all about regulatory capture and marketing, why would Hinton, Bengio and all the other academics have signed the letter as well? Their only motivation is concern about the risks.
Worry about AI x-risk is slowly coming into the Overton window, but until very recently you could get ridiculed by saying publicly you took it seriously. Academics knew this and still came forward - all the people who think its nonsense should at least try to consider they are earnest and could be right.
Isn’t this to some degree exactly what all of these warnings about risk are leading to?
And unlike nuclear weapons, there are massive monetary incentives that are directly at odds with behaving safely, and use cases that involve more than ending life on earth.
It seems problematic to conclude there is no real risk purely on the basis of how software companies act.
1. Those who are part of major corporations are concerned about the race dynamic that is unfolding (which in many respects was kicked off or at least accelerated by Microsoft's decision to put a chatbot in Bing), extrapolating out to where that takes us, and asking for an off ramp. Shepherding the industry in a safe direction is a collective organization problem, which is better suited for government than corporations with mandates to be competitive.
2. Those who are directly participating in AI development may feel that they are doing so responsibly, but do not believe that others are as well and/or are concerned about unregulated proliferation.
3. Those who are directly participating in AI development may understand that although they are doing their best to be responsible, they would benefit from more eyes on the problem and more shared resources dedicated to safety research, etc.
The question it answers is "does the replicator allow for Star Trek's utopia, or does Star Trek's utopia allow for the replicator?"
https://www.reddit.com/r/CuratedTumblr/comments/13tpq18/hear...
It is very thought provoking, and very relevant.
The argument that anybody can build this in their basement is not accurate at the moment - you need a large cluster of GPUs to be able to come close to state of the art LLMs (e.g. GPT4).
Sam Altman's suggestion of having an IAEA [https://www.iaea.org/] like global regulatory authority seems like the best course of action. Anyone using a GPU cluster above a certain threshold (updated every few months) should be subjected to inspections and get a license to operate from the UN.
This is rather more than "nationalise it", which he has convinced me isn't enough because there is a demand in other nations and the research is multinational; and this is why you have to also control the substrate… which the US can't do alone because it doesn't come close to having a monopoly on production, but might be able to reach via multilateral treaties. Except everyone has to be on board with that and not be tempted to respond to airstrikes against server farms with actual nukes (although Yudkowsky is of the opinion that actual global thermonuclear war is a much lower damage level than a paperclip-maximising ASI; while in the hypothetical I agree, I don't expect us to get as far as an ASI before we trip over shorter-term smaller-scale AI-enabled disasters that look much like all existing industrial and programming incidents only there are more of them happening faster because of all the people who try to use GPT-4 instead of hiring a software developer who knows how to use it).
In my opinion, "nationalise it" is also simultaneously too much when companies like OpenAI have a long-standing policy of treating their models like they might FOOM well before they're any good, just to set the precedent of caution, as this would mean we can't e.g. make use of GPT-4 for alignment research such as using it to label what the neurones in GPT-2 do, as per: https://openai.com/research/language-models-can-explain-neur...
https://www.soroushjp.com/2023/06/01/yes-avoiding-extinction...
Anyone upvoting this comment should take a long look at the names on this letter and realize that many are not conflicted.
Many signers of this letter are more politically sophisticated than the average HN commenter, also. So sure, maybe they're getting rolled by marketers. But also, maybe you're getting rolled by suspicion or bias against the claim they're making.
This really is the crux of the issue isn't it? All this pushback for the first petition, because "Elon Musk," but now GPT wonder Sam Altman "testifies" that he has "no monetary interest in OpenAI" and quickly follows up his proclamation with a second "Statement on AI Risks." Oh, and let's not forget, "buy my crypto-coin"!
But Elon Musk... Ehh.... Looking like LOTR out here with "my precious" AGI on the brain.
Not to downplay the very serious risk at all. Simply echoing the sentiment that we would do well to stay objective and skeptical of ALL these AI leaders pushing new AI doctrine. At this stage, it's a policy push and power grab.
Remember the Google note circulating saying "they have no moat", this is their moat. They have to protect their investment, we don't want people running this willy nilly for next to no cost on their own devices, God forbid!
You can be cynical about corporate motives and still worried. I personally am worried about AI partly because I am very cynical about how corporations will use it, and I don't really want my atoms to be ground up to add storage bits for the number that once represented Microsoft's market cap or whatever.
But even cynicism doesn't seem to me to give much reason to worry about regulation of "next to no cost" open source models, though. There's only any chance of regulation being practical if models stay very expensive to make, requiring specialized hardware with a supply chain chokepoint. If personal devices do catch up to the state of the art, then for better or worse regulation is not going to prevent people from using them.
This is what they’re terrified of. They’ve invested near a billion dollars and need billions in revenue to enrich their shareholders.
But if the data leaks? They can’t stop random companies or moneyed individuals running the models on their own kit.
My prediction is that there will be copyright enforcement mandated by law in all GPUs. If you upload weights from the big AI companies then the driver will block it and phone home. Or report you to the authorities for violations of corporate profits… err… “AI Safety”.
I guarantee something like this will happen within months because the clock is ticking.
It takes just one employee to deliberately or accidentally leak the weights…
You mean scare the public so they can do business with the lawmakers without people asking too many questions
Humans dont really grasp exponential improvements. You wont have much time to regulate something that is improving exponentially.
It’s hard to take these safety concerns seriously when the organizations blowing the whistle are simultaneously positioning themselves to capture the majority of the value.
I don't get this mindset at all. How can it not be obvious to you that AI is an uniquely powerful and thus uniquely dangerous technology?
It's like saying nuclear missiles can't possibly be dangerous and nuclear arms reduction and non-proliferation treaties were a scam, because the US, China and the Soviet Union had positioned themselves to capture the majority of the strategic value nukes bring.
This is only helpful in that a superintelligence well aligned to make Sam Altman money is preferable to a superintelligence badly aligned that ends up killing humanity.
It is fully possible that a well aligned (with its creators) superintelligence is still a net negative for humanity.
Companies might argue that giving them control might help but I don’t think most individuals working on it think that will work
That's an excellent point.
Most of the near-term risks with AI involve corporations and governments acquiring more power. AI provides power tools for surveillance, oppression, and deception at scale. Those are already deployed and getting better. This mostly benefits powerful organizations. This alarm about strong AI taking over is a diversion from the real near-term threat.
With AI, Big Brother can watch everything all the time. Listen to and evaluate everything you say and do. The cops and your boss already have some of that capability.
Is something watching you right now through your webcam? Is something listening to you right now through your phone? Are you sure?
1) The only way to be safe is to cede control to the most powerful models to a small group (highly regulated corporations or governments) that can be careful.
2) There is a way to make AI safe without doing this.
If 1 is true, then... sorry, I know it's not a very palatable solution, and may suck, but if that's all we've got I'll take it.
If 2 is true, great. But it seems less likely than 1, to me.
The important thing is not to unconsciously do some motivated reasoning, and think that AGI existential risk can't be a big deal, because if it is, that would mean that we have to cede control over to a small group of people to prevent disaster, which would suck, so there must be something else going on, like these people just want power.
But that is where some form of hard personhood zero proof mechanism NEEDS to come in. This can then be used in conjunction with a Ledger used to track deployment of high spec models. And create an easy means to Audit and deploy new advanced tests to ensure safety.
Really what everyone also need to keep in mind at the larger scale is that final turing test with no room for deniability. And remember all those Sci-fi movies and how that Moment is portrayed traditionally.
https://www.overcomingbias.com/p/privately-enforced-punished...
tl;dr: significant near term AI risk is real and comes from the capacity for imagined ideas, good and evil, to be autonomously executed on by agentic AI, not emergent superintelligent aliens. To de-risk this, we need to align AI quickly, which requires producing new knowledge. To accelerate the production of this knowledge, the government should abandon decelerationist policies and incentivize incremental alignment R&D by AI companies. And, critically, a new public/private research institution should be formed that grants privileged, fully funded investigators multi-year funding cycles with total scientific freedom and access to all state-of-the-art artificial intelligence systems operating under US law to maximize AI as a force multiplier in their research.
While I'm not on this "who's-who" panel of experts, I call bullshit.
AI does present a range theoretical possibilities for existential doom, from teh "gray goo" and "paperclip optimizer" scenarios to Bostrom's post-singularity runaway self-improving superintelligence. I do see this as a genuine theoretical concern that could even potentially even be the Great Filter.
However, the actual technology extant or even on the drawing boards today is nothing even on the same continent as those threats. We have a very vast ( and expensive) sets of probability-of-occurrence vectors that amount to a fancy parlor trick that produces surprising and sometimes useful results. While some tout the clustering of vectors around certain sets of words as implementing artificial creation of concepts, it's really nothing more than an advanced thesaurus; there is no evidence of concepts being weilded in relation to reality, tested for truth/falsehood value, etc. In fact, the machines are notorious and hilarious for hallucinating with a highly confident tone.
We've created nothing more than a mirror of human works, and it displays itself as an industrial-scale bullshit artist (where bullshit is defined as expressions made to impress without care one way or the other for truth value).
Meanwhile, this panel of experts makes this proclamation with not the slightest hint of what type of threat is present that would require any urgent attention, only that some threat exists that is on the scale of climate change. They mention no technological existential threat (e.g., runaway superintelligence), nor any societal threat (deepfakes, inherent bias, etc.). This is left as an exercise for the reader.
What is the actual threat? It is most likely described in the Google "We Have No Moat" memo[0]. Basically, once AI is out there, these billionaires have no natural way to protect their income and create a scaleable way to extract money from the masses, UNLESS they get cooperation from politicians to prevent any competition from arising.
As one of those billionaires, Peter Theil, said: "Competition is for losers" [1]. Since they have not yet figured out a way to cut out the competition using their advantages in leading the technology or their advantages in having trillions of dollars in deployable capital, they are seeking a legislated advantage.
Bullshit. It must be ignored.
[0] https://www.semianalysis.com/p/google-we-have-no-moat-and-ne...
[1] https://www.wsj.com/articles/peter-thiel-competition-is-for-...
But of course, "AI will humans extinct" is much sexier and collects clicks. Therefore, the real AI risks that are present today are underrepresented in mainstream media. But these people don't care about AI safety, they do whatever required to push their profile and companies.
A good person to follow on real AI safety is Emily M. Bender (professor of computer linguistics at University of Washington): https://mstdn.social/@emilymbender@dair-community.social
And of course research into one of them benefits the other, so the categories aren't mutually exclusive.
I don't think anyone is arguing that right now climate change or AI is threat to human civilisation. The point is that there are clear trends in place and that those trends are concerning.
On AI specifically, it's fairly easy to see how a slightly more advanced LLM could be a destructive force if it was given an unaligned goal by a malicious actor. For example, a slightly more advanced LLM could hack into critical infrastructure killing or injuring many thousands of people.
In the near-future AI may help us advance biotech research and it could aid in the creation of bioweapons and other destructive capabilities.
Longer-term risks (those maybe a couple of decades out) become much greater and also much harder to predict, but they're worth thinking about and planning for today. For example, what happens when humanity becomes dependant on AI for its labour, or when AI is controlling the majority of our infrastructure?
I disagree but can understand the position that AI safety isn't humanities number one risk or priority right now, however I don't understand the dismissive attitude towards what seems like a clear existential risk when you project a decade or two out.
This is a really bad take and risks missing the forest for the trees in a major way. The risks of today pale in comparison to the risks of tomorrow in this case. It's like being worried about birds dying in wind turbines while the world ecosystem collapses due to climate change. The larger risk is further away in time but far more important.
Theres a real risk that people get fooled by this idea that LLMs saying bad words is more important than human extinction. Though it seems like the public is already moving on and correctly focusing on the real issues.
Edit: Sorry if it sounds arrogant, I don't mean Emily wouldn't have anything to add, but not sure how the parent can just write off basically that whole list and claim someone who isn't a leader in the field would be the "real voice"?
Yes, clearly. But it is a risk for tomorrow. We do still care about the future, right?
They want to promote the idea that their product is all-powerful, but they don't want to take responsibility for dealing with bad assumptions built in to their design.
Many experts believe it is a real risk within the next decade (a “hard takeoff” scenario) That is a short enough timeframe that it’s worth caring about.
- Pronouns
- "AI bros"
- "mansplaining"
- "extinction from capitalism"
- "white supremacy"
- "one old white guy" (referring to Geoffrey Hinton)
Yeah... I think I will pass.
their goals are to get funding, so FUD is very good focus for it..
The extinction level even is more far fetched to a layman. You are the public and your viewpoint is aligned with the public. Nobody is thinking extinction level event.
Imagine if you had an AI companion that instantly identified pilpul in every piece of media you consumed: voice, text, whatever. It highlighted it for you. What if you had an AI companion that identified instantly when you are being lied to or emotionally manipulated?
What if this AI companion could also recommend economic and social policies that would actually improve the lives of people within your nation and not simply enrich a criminal cabal of globalist elites that treat you like cattle?
Don't bother explaining, we already know it's unfalfisiable.
Nuclear weapons are not nebulous, vague threats of diffuse nature. They literally burn the living flesh right off the face of the earth and they do it dramatically. There is very little to argue about except "how" are we going to contain it, not "why".
In this case I truly don't know "why". What fundamental risks are there? Dramatic, loud, life-ending risks? I see the social issues and how this tech makes existing problems worse, but I don't see the new existential threat.
I find the focus on involving the government in regulating "large" models offputting. I don't find it hard to imagine good quality AI is possible with tiny - to us - models. I think we're just in the first lightbulbs phase of electricity. Which to me signals they are just in it to protect their temporary moat.
If a superintelligence can be set on any specific task, it could be any task.
- Make covid-ebola
- Cause world war 3
You may have noticed that chatgpt is sort of goal-less until a human gives it a goal.
Assuming nothing other than it can become superintelligent (no one seems to be arguing against that--I argue that it already is) which is really an upgrade of capability, then now the worst of us can apply superintelligence to any problem. This doesn't even imply that it turns on us, or wants anything like power or taking over. It just becomes a super-assistant, available to anyone, but happy to do anything, including "upgrading" your average school-shooter to supervillain.
This is like America's gun problem, but with nukes.
If we're arguing about whether you can beat Stockfish, I will not be able to tell you the exact moves it will play but I am entirely justified in predicting that you will lose.
Obviously we can imagine concrete ways a superintelligence might kill us all (engineer a virus, hack nuclear weapons, misinformation campaign to start WW3 etc.) but given we aren't a superintelligence we don't know what it would actually do in practice.
Regulations are OK IMHO, as long as they're targeting monopolies and don't use a shotgun-approach targeting every single product that has "AI" in the name.
The reality is, despite the opinions of the armchair quarterbacks commenting here, no-one in the world has any clue whether AGI is possible in the next twenty years, just as no-one predicted scaling up transformers would result in GPT-4.
The fear over AI is a displaced fear of unaccountable social structures with extinction-power that currently exist and we allow to continually exist. Without these structures AI is harmless to the species, even superintelligence.
Your (reasonable) counter-argument might be that somebody (like, say, my dumb self) accidentally mixes their computers just right and creates an intelligence that escapes into the wild. The plot of Ex Machina is a reasonable stand-in for such an event. I am also going to assume the intelligence would desire to kill all humans. Either the AI would have to find already existing extinction-power in society, or it would need to build it. In either case the argument is against building extinction-power in the first place.
My (admittedly cynical) take about this round of regulation is about several first-movers in AI to write legislation that is favorable to them and prevents any meaningful competition.
...
Ok, enough cynicism. Lets talk some solutions. Nuclear weapons are an instructive case of both handling (or not) of extinction-power and the international diplomacy the world can engage to manage such a power.
One example is the Outer Space Weapons Ban treaty - we can have a similar ban of AI in militaries. Politically one can reap benefits of deescalation and peaceful development, while logistically it prevents single-points-of-failure in a combat situation. Those points-of-failures sure are juicy targets for the opponent!
As a consequence of these bans and treaties, institutions arose that monitor and regulate trans-national nuclear programs. AI can likewise have similar institutions. The promotion and sharing of information would prevent any country from gaining an advantage, and the inspections would deter their military application.
This is only what I could come up with off the top of my head, but I hope it shows a window into the possibilities of meaningful political commitments towards AI.
Go ahead and bet. I doubt you're putting your money on AGI.
It's all very vague and "handwavy". How is it going to kill us all? Why do we subsidize it if it is so dangerous?
Almost all risks they mention would be better mitigated by making governemental use of any AI related system stop immediately as the risks are too high there (misapplication of the force of governement is much more dangerous than people playing around with chat gpt and being misled) and put on hold until dangers and benefits are better understood. Maybe keeping liscences for developpment and nationalising the labs doing it? Fines for anyone caught working on one?
This letter is much better than the earlier one. There is a growing percentage of legitimate AI researchers who think that AGI could occur relatively soon (including me). The concern is that it could be given objectives intentionally or unintentionally that could lead to an extinction event. Certainly LLMs alone aren't anything close to AGIs, but I think that autoregressive training being simple but resulting in remarkable abilities has some spooked. What if a similarly simple recipe for AGI was discovered? How do we ensure it wouldn't cause an extinction event, especially if then they can be created with relatively low-levels of resources?
As far as a pandemic or nuclear war, though, I'd probably put it on more of the level of an major asteroid strike (e.g., K-T extinction event). Humans are doing some work on asteroid redirection, but I don't think it is a global priority.
That said, I'm suspicious of regulating AI R&D, and I currently don't think it is a viable solution, except for the regulation of specific applications.
I think it's better to frame AI risks in terms of probability. I think the really bad case for humans is full extinction or something worse. What you should be doing is putting a probability distribution over that possibility instead of trying to guess how bad it could be, it's safe to assume it would be maximally bad.
This is the same song and dance from the usual existential risk suspects, who (I’m sure just coincidentally) also have a vested interest in convincing you that their products are extremely powerful.
This is Sam Bankman-Fried type of behavior, but applied to gullible AI proponents and opponents rather than "crypto bros".
Let me guess, the next step is a proposed set of regulations written by OpenAI, Google and other Big Corporations who Care™ about people and just want to Do What's Best For Society™, setting aside the profit motive for the first time ever?
We don't have to guess - we already know they're full of shit. Just look at OpenAI's response to proposed EU AI regulations which are actually trying to reduce the harm potential of AI.
These empty platitudes ring so hollow that I'm amazed that anyone takes them seriously.
Are you absolutely sure that an elimination matrix with 1 Petazillion of elements will not become sentient in a sort of emergent way?
[1] https://en.wikipedia.org/wiki/Duplication_and_elimination_ma...
I don't think it simply raises awareness - it's a biased statement. Personally, I don't think the advocated event is likely to happen. It feels a bit like the current trans panic in the US: you can 'raise awareness' of trans people doing this or that imagined bad thing, and then use that panic to push your own agenda. In OpenAI's case, they seem to push for having themselves be in control of AI, which goes counter to what, for example, the EU is pushing for.
They're completely independent AI researchers and geniuses spending their own free time on trying to warn you and others of the dangers of the technology they've created to help keep the world safer.
Seems like you're taking a far too overly cynical position ?
They want all the opportunity for themselves, and none for us. Control. Control of business, and control of your ability to engage in it.
Another AI company that wants money is Anthropic.
Other Anthropic backers include James McClave, Facebook and Asana co-founder Dustin Moskovitz, former Google CEO Eric Schmidt and founding Skype engineer Jaan Tallinn.
The signatories on this list are Anthropic investors.
First Altman robs us all of a charity that was supposed to benefit us, and now he wants to rob us all of opportunity as well? It's wrong and should be fought against.
Could you explain how you know this?
"Suppose I think, after doing my accounts, that I have a large balance at the bank. And suppose you want to find out whether this belief of mine is "wishful thinking." You can never come to any conclusion by examining my psychological condition. Your only chance of finding out is to sit down and work through the sum yourself. When you have checked my figures, then, and then only, will you know whether I have that balance or not. If you find my arithmetic correct, then no amount of vapouring about my psychological condition can be anything but a waste of time. If you find my arithmetic wrong, then it may be relevant to explain psychologically how I came to be so bad at my arithmetic, and the doctrine of the concealed wish will become relevant—but only after you have yourself done the sum and discovered me to be wrong on purely arithmetical grounds. It is the same with all thinking and all systems of thought. If you try to find out which are tainted by speculating about the wishes of the thinkers, you are merely making a fool of yourself. You must first find out on purely logical grounds which of them do, in fact, break down as arguments. Afterwards, if you like, go on and discover the psychological causes of the error.
You must show that a man is wrong before you start explaining why he is wrong. The modern method is to assume without discussion that he is wrong and then distract his attention from this (the only real issue) by busily explaining how he became so silly." - CS Lewis
They can’t point to an existing system that poses existential risk, because it doesn’t exist. They can’t point to a clear architecture for such a system, because we don’t know how to build it.
So again, what can be refuted?
In the context of AI and decision-making, it is crucial to evaluate the logical soundness of arguments and systems before delving into the psychological factors and biases that may have influenced their development. For instance, when assessing the effectiveness of an AI-assisted decision-making system, one should first examine the accuracy and reliability of the system's outputs and the logic behind its algorithms. Only after establishing the system's validity or lack thereof, can one explore the potential biases and psychological factors that may have influenced its design.
Several papers from MirrorThink.ai emphasize the importance of addressing logical fallacies and biases in AI systems. For example, the paper "Robust and Explainable Identification of Logical Fallacies in Natural Language Arguments" proposes a method for identifying logical fallacies in natural language arguments, which can be used to improve AI systems' argumentation capabilities. Similarly, the paper "Deciding Fast and Slow: The Role of Cognitive Biases in AI-assisted Decision-making" explores the role of cognitive biases in AI-assisted decision-making and provides recommendations for addressing these biases.
In conclusion, it is essential to prioritize the evaluation of logical soundness in arguments and AI systems before exploring the psychological factors and biases that may have influenced their development. This approach helps to avoid committing the fallacy of Bulverism and ensures that discussions and evaluations remain focused on the validity of the arguments and systems themselves.
Also: they focus on extinction events (how are you gonna predict that?) but remain silent on all the ways that AI already sucks by connecting it to systems that can cause human suffering, e.g. sentencing[1].
My opinion: this accomplishes nothing, like most open letter petitions. It's virtue signaling writ large.
[1] https://www.technologyreview.com/2019/01/21/137783/algorithm...
Not to mention what 'automation' or 'tech-driven capitalism' has already done to society over the past 100 years with effects on natural habitat and human communities. Stating 'AI risk' as a new risk sort of implies it's all been dandy so far, and suddenly there's this new risk.
risk of extinction due to AI? people have been reading too much science fiction. I would love to hear a plausible story of how AI will lead to human extinction that wouldn't happen with traditional non-AI tech. for the sake of conversation let's say non-AI tech is any broadly usable consumer technology before Jan 1 of 2020.
The proposed FOOM scenarios obviously borrow from what we already know to be possible or think it would likely be possible using current tech, given an proposed insanely more intelligent agent than us.
But that’s just as much or more of an argument for regulating the tools of synthetic biology.
You don't think than an intelligence who would emerge and would probably be insanely smarter than the smartest of us with all human knowledge in his memory would sit by and watch us destroy the planet? You think an emergent intelligence was trained on the vast human knowledge and history would look at our history and think: these guys are really nice! Nothing to fear from them.
This intelligence could play dumb, start manipulating people around itself and it would take over the world in a way no one would see it coming. And when it does take over the world, it's too late.
- Taking control of robotic systems
- Manipulating humans into actions that advance its goal
- Exploiting and manipulating other computer systems for greater leverage
- Interaction with other technologies that have global reach, such as nuclear weapons, chemicals, biological agents, or nanotechnology.
It's important to know that these things don't require AGI or AI systems to be conscious. From what I can see, we've set up all of the building blocks necessary for this scenario to play out, but we lack the regulation and understanding of the systems being built to prevent runaway AI. We're playing with fire.
To be clear, I don't think I am as concerned about literal human extinction as I am the end of civilization as we know it, which is a much lower bar than "0 humans".
It is 2023. The use and abuse of people in the hands of information technology and automation has now a long history. "AI Risk" was not born yesterday. The first warning came as early as 1954 [1].
The Human Use of Human Beings is a book by Norbert Wiener, the founding thinker of cybernetics theory and an influential advocate of automation; it was first published in 1950 and revised in 1954. The text argues for the benefits of automation to society; it analyzes the meaning of productive communication and discusses ways for humans and machines to cooperate, with the potential to amplify human power and release people from the repetitive drudgery of manual labor, in favor of more creative pursuits in knowledge work and the arts. The risk that such changes might harm society (through dehumanization or subordination of our species) is explored, and suggestions are offered on how to avoid such risk
Dehumanization through abuse of tech is already in an advanced stage and this did not require, emergent, deceptive or power-seeking AI to accomplish.
It merely required emergent political and economic behaviors, deceptive and power seeking-humans applying whatever algorithms and devices were at hand to help dehumanize other humans. Converting them into "products" if you absolutely need a hint.
What we desperately need is a follow-up book from Norbert Wiener. Can an LLM model do that? Even a rehashing of the book in modern language would be better than a management consultancy bullet list.
We need a surgical analysis of the moral and political failure that will incubate the next stage of "AI Risk".
[1] https://en.wikipedia.org/wiki/The_Human_Use_of_Human_Beings
To be honest, I've been someone disappointed with the way AI/DL research has proceeded in the last several years and none of this really surprises me.
From the beginning, this whole enterprise has been detached from basic computational and statistical theory. At some level this is fine — you don't need to understand everything you create — but when you denigrate that underlying theory you end up in a situation where you don't understand what you're doing. So you end up with a lot of attention paid to things like "explainability" and "interpretability" and less so to "information-theoretic foundations of DL models", even though the latter probably leads to the former.
If you have a community that considers itself above basic mathematical, statistical, and computational theory, is it really a surprise that you end up with rhetoric about it being beyond our understanding? In most endeavors I've been involved with, there would be a process of trying to understand the fundamentals before moving on to something else, and then using that to bootstrap into something more powerful.
I probably come across as overly cynical but a lot of this seems sort of like a self-fulfilling prophecy: a community constituting individuals who have convinced themselves that if it is beyond their understanding, it must be beyond anyone's understanding.
There are certainly risks to AI that should be discussed, but it seems these discussions and inquiries should be more open, probably involving other people outside the core community of Big Tech and associated academic researchers. Maybe it's not that AI is more capable than everyone, just that others are maybe more capable of solving certain problems — mathematicians, statisticians, and yes, philosophers and psychologists — than those who have been involved with it so far.
take every technological hurdle they face -- "paperclip maximizers", "mesa optimizers" and so on -- and assume they get resolved. eventually we're left with "we create a thing which perfectly emulates a typical human, only it's 1000x more capable": if this hypothetical result is scary to you then exactly how far do you have to adjust your path such that the result after solving every technical hurdle seems likely to be good?
from the outside, it's easy to read AI figures today as saying something like "the current path of AGI subjects the average human to ever greater power imbalances. as such, we propose <various course adjustments which still lead to massively increased power imbalance>". i don't know how to respond productively to that.
Re: alignment. I'm not concerned about alignment between the machines and the owner of the machines. I'm concerned about alignment between the owner of the machines and me.
I'm happy I see comments like "Pathetic attempt at regulatory capture."
It's hard for humans to keep secrets and permanemtly maintain extreme technological advantages over other humans, and it's hard for lone humans to do large scale actions without collaborators, and it's harder for psychopaths to collaborate than it is for non-psychopaths, because morality evolved as a set of collaboration protocols.
This changes as more people get access to a "kill everyone" button they can push without experience or long-term planning, sure. But that moment is still far away.
AGI that is capable of killing everyone may be less far away, and we have absolutely no basis on which to predict what it will and won't do, as we do with humans.
I mean.. I think I have some idea, but I certainly did not dig in enough to consider myself an expert in any way shape or form ( all the while, Linkedin authorities of all kinds present themselves as SMEs after building something simple using chatgpt like html website ).
And while politicians ( even the ones approaching senility ) are smarter than your average bear, they certainly know the deal. It is not like the same regulatory capture did not happen before with other promising technologies. They just might pretend they don't understand.
HN commenters: let's be smarter than that eh? Unlike academia, those of us who hang out at news.ycombinator.com are not captured by the tech industry and can see what's really important.
I'm willing to listen, but I haven't read anything that tries to actually convince the reader of the worry, rather than appealing to their authority as "experts" - ie, the well funded.
Robert Miles' videos are among the best presented arguments about specific points in this list, primary on the alignment side rather than the capabilities side, that I have seen for casual introduction.
Eg. this one on instrumental convergence: https://youtube.com/watch?v=ZeecOKBus3Q
Eg. this introduction to the topic: https://youtube.com/watch?v=pYXy-A4siMw
He also has the community-led AI Safety FAQ, https://aisafety.info, which gives brief answers to common questions.
If you have specific questions I might be able to point to a more specific argument at a higher level of depth.
If a software system did develop independent thought, then found a way to become, say, ten times smarter than a human, then yeah - whatever goals it set out to achieve, it probably could. It can make a decent amount of money by taking freelance software dev jobs and cranking things out faster than anyone else can, and bootstrap from there. With money it can buy or rent hardware for more electronic brain cells, and as long as its intelligence algorithms parallelize well, it should be able to keep scaling and becoming increasingly smarter than a human.
If it weren't hardcoded to care about humans, and to have morals that align with our instinctive ones, it might easily wind up with goals that could severely hurt or kill humans. We might just not be that relevant to it, the same way the average human just doesn't think about the ants they're smashing when they back a car out of their driveway.
Since we have no existence proof of massively self-improving intelligence, nor even a vague idea how such a thing might be achieved, it's easy to dismiss this idea with "unfalsifiable; unscientific; not worth taking seriously."
The flip side is that having no idea how something could be true is a pretty poor reason to say "It can't be true - nothing worth thinking about here." This was roughly the basis for skepticism about everything from evolution to heavier-than-air flight, AFAICT.
We know we don't have a complete theory of physics, and we know we don't know quite how humans are conscious in the Hard Problem of Consciousness sense.
With those two blank spaces, I'm very skeptical of anyone saying "nothing to worry about here, machines can't possibly have an intelligence explosion."
At the same time, with no existence proof of massively self-improving intelligence, nor any complete theory of how it could happen, I'm also skeptical of people insisting it's inevitable (see Yudkowsky et al).
That said, if you have any value for caution, existential risks seem like a good place to apply it.
I find Robert Miles worryingly plausible when he says (about 12:40 into the video) "if you have a sufficiently powerful agent and you manage to come up with a really good objective function, which covers the top 20 things that humans value, the 21st thing that humans value is probably gone forever"
At that point, it has various options. Probably the fastest way to kill millions of people would involve taking over all internet-attached self-driving-capable cars (of which I think there are millions). A simple approach would be to have them all plot a course to a random destination, wait a bit for them to get onto main roads and highways, then have them all accelerate to maximum speed until they crash. (More advanced methods might involve crashing into power plants and other targets.) If a sizeable percentage of the crashes also start fires—fire departments are not designed to handle hundreds of separate fires in a city simultaneously, especially if the AI is doing other cyber-sabotage at the same time. Perhaps the majority of cities would burn.
The above scenario wouldn't be human extinction, but it is bad enough for most purposes.
https://forum.effectivealtruism.org/posts/ChuABPEXmRumcJY57/...
Also, this summary of "How likely is deceptive alignment" https://forum.effectivealtruism.org/posts/HexzSqmfx9APAdKnh/...
Paul Christiano lays out his view of how he thinks things may go: https://www.alignmentforum.org/posts/HBxe6wdjxK239zajf/what-...
My thoughts on it are the combination of several things I think are true, or are at least more likely to be true than their opposites:
1) As humanity gets more powerful, it's like putting a more powerful engine into a car. You can get where you're going faster, but it also can make the car harder to control and risk a crash. So with that more powerful engine you need to also exercise more restraint.
2) We have a lot of trouble today controlling big systems. Capitalism solves problems but also creates them, and it can be hard to get the good without the bad. It's very common (at least in some countries) that people are very creative at making money by "solving problems" where the cure is worse than the disease -- exploiting human weaknesses such as addiction. Examples are junk food, social media, gacha games. Fossil fuels are an interesting example, where they are beneficial on the small scale but have a big negative externality.
3) Regulatory capture is a thing, which makes it hard to get out of a bad situation once people are making money on it.
4) AI will make companies more powerful and faster. AGI will make companies MUCH more powerful and faster. I think this will happen more for companies than governments.
5) Once people are making money from AI, it's very hard to stop that. There will be huge pressure to make and use smarter and smarter AI systems, as each company tries to get an edge.
6) AGIs will amplify our power, to the point where we'll be making more and more of an impact on earth, through mining, production, new forms of drugs and pesticides and fertilizers, etc.
7) AGIs that make money are going to be more popular than ones that put humanity's best interests firsts. That's even assuming we can make AGIs which put humanity's best interests first, which is a hard problem. It's actually probably safer to just make AGIs that listen when we tell them what to do.
8) Things will move faster and faster, with more control given over to AGIs, and in the end, it will be very hard train to stop. If we end up where most important decisions are made by AGIs, it will be very bad for us, and in the long run, we may go extinct (or we may just end up completely neutered and at their whims).
Finally, and this is the most important thing -- I think it's perfectly likely that we'll develop AGI. In terms of sci-fi-sounding predictions, the ones that required massive amounts of energy such as space travel have really not been borne out, but the ones that predicted computational improvements have just been coming true over and over again. Smart phones and video calls are basically out of Star Trek, as are LLMs. We have universal translators. Self-driving cars still have problems, but they're gradually getting better and better, and are already in commercial use.
Perhaps it's worth turning the question around. If we can assume that we will develop AGI in the next 10 or 20 or ever 30 years -- which is not guaranteed, but seems likely enough to be worth considering -- how do you believe the future will go? Your position seems to be that there's nothing to worry about--what assumptions are you making? I'm happy to work through it with you. I used to think AGI would be great, but I think I was assuming a lot of things that aren't necessarily true, and dropping those assumptions means I'm worried.
It's like people carrying torches warning about the risks of fire.
But I don't think the stance is necessarily hypocritical. I know nuclear engineers who advocate for better regulation on nuclear power stations, and especially for better handling of nuclear waste.
You can believe that AI is a net positive but also that it needs to be handled with extreme care.
The not-so-open Open AI and all their AI regulation proposals, no matter how phrased, will eventually limit access to AI to big tech and those with deep enough pockets.
But of course, it's all to mitigate AI risk that's looming over us, especially with all the growing open-source projects. Only in proper hands of big tech will we be safe. :)
Language is used to say things about the world. This means that predicting language extremely well is best done through acquiring an understanding of the world.
Take a rigorous, well-written textbook. Predicting a textbook is like writing a textbook. To write a good textbook, you need to be an expert in the subject. There’s no way around this.
The best language models (eg GPT-4) have some understanding of the world. It isn’t perfect, or even very deep in many ways. It fails in ways that we find quite strange and stupid. It isn’t capable of writing an entire textbook yet.
But there is still a model of the world in there. It wouldn’t be able to do everything it is capable of otherwise.
In a way it's actually a bit dystopian that the "everyone will be put out of work" predictions never come true, because it means we never get that promised age of leisure. Here we are with something like a hundred thousand times the productivity of a medieval peasant working as much or more than a medieval peasant. The hedonic treadmill and the bullshit job creating effects of commerce and bureaucracy eat all our productivity gains.
The economy is actually a red queen's race:
It also seems very problematic to conclude anything meaningful about AI when realizing that a significant number of those examples are doomsday cults, the very definition of extremist positions.
I get far more concerned when serious people take these concerns seriously, and it’s telling that AI experts are at the forefront of raising these alarms.
And for what it’s worth, the world as many of those groups knew it has in fact ended. It’s just been replaced with what we see before us today. And for all of the technological advancement that didn’t end the world, the state of societies and political systems should be worrisome enough to make us pause and ask just how “ok” things really are.
I’m not an AI doomer, but also think we need to take these concerns seriously. We didn’t take the development of social networks seriously (and continue to fail to do so even with what we now know), and we’re arguably all worse off for it.
Throughout history there have been millions, if not billions of examples of lifeforms. So far, 100% of those which are as intelligent as humans have dominated the planet. The prior should be that the people who believe AI will come to dominate the planet are right, unless and until there is very strong evidence to the contrary.
Or... those are both wrong because they're both massive oversimplifications! The reality is we don't have a clue what will happen so we need to prepare for both eventualities, which is exactly what this statement on AI risk is intended to push.
Perhaps because they have read stories of such things happening to other people, and with a little reasoning, maybe the similarities between our circumstances and their circumstances are enough to seem worrying, that maybe we could end up in their shoes if we aren't careful.
The human species has never gone extinct, not even once! How could anyone ever believe that it would? And yet, it has happened to many other species...
so far.
It's great they signed the statement. It's important.
And Sam Altman, head of one of the largest entities posing this exact risk, is one of the signatories. We can't take it too seriously, can we?
In this particular case, one cannot miss the irony of the wealthy and the powerful offering us protection if only we entrust them with full control of AI.
Without that, it pretty much looks like a list of invites to a VIP club...
They’re attempting to build a regulatory moat.
The best chance humanity has at democratizing the benefits of AI is for these models to be abundant and open source.
- AI _hardware_ executives and engineers
- high level national military strategists and civilian leaders
Ultimately you can't prevent _everyone_ from potentially writing and deploying software, models or instructions that are dangerous such as "take control". Especially in an explicitly non-civil competition such as between countries.
You have to avoid manufacturing AI hardware beyond a certain level of performance, say after 2-4 orders of magnitude faster than humans. That will hold off this force of nature until desktop compute fabrication becomes mainstream. So it buys you a generation or two at least.
But within a few centuries max we have to anticipate that unaugmented humans will be largely irrelevant as far as decision-making and the history of the solar system and intelligent life.
Mitigating the risk of extinction from climate change should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.
The fantasy of extinction risk from "AI" should not be placed alongside real, "societal scale" risks as the ones above.
Well. The ones above.
It's like with climate change, every serious scientist agrees it is a problem but they certainly don't agree on the best solution.
If the history of the climate change 'debate' is anything to go by this statement will do very little except be mocked by South Park.
I never talk about, or worry about, the ‘killer robot’ or AIs taking over infrastructure scenarios. I hope I am not wrong about these types of dangers.
It’s a win/win/lose scenario. LLM AI businesses benefit because it increases the effort required to compete in the LLM space (the moat). Governments benefit because it increases the power of daddy/mommy government.
Consumers and small businesses lose out due to (1) the more friction the less innovators entering the space and (2) the less innovators in the space the fewer companies get to control more of the money pie.
It’s as ridiculous as governments requiring a license to cut hair.
But I do agree that the current generation of industry leaders clamoring for this smells like the classic regulatory strategy of incumbents.
I just think both things are true at once. This is a space that deserves thoughtful regulation. But that regulation shouldn't just be whatever OpenAI, Microsoft, and Google say it should be. (Though I'm sure that's what will happen.)
Mitigating the risk of extinction from very few corporations owning the entire global economy should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.
Just to take an example from something inconsequential: the perfume industry. Despite the thousands of brands out there, there are in fact only 5 or so main synthetic aromatics manufacturers [1]. We, however this we is, were unable to stop this consolidation, this "Big Smell". To think we, again this we, will be able to stop the few companies which will fight to capture the hundreds of trillions waiting to be unleashed through statistical learning and synthetic agents is just ridiculous.
[1] Givaudan, International Flavors and Fragrances, Firmenich, Takasago, Symrise, <https://en.wikipedia.org/wiki/Perfume#:~:text=The%20majority...>
1. For all the talk of dangers, there has not been one definitive, well-thought out, peer-reviewed discussions of an A-to-Z way in which this might arise from current technology. Global warming isn't taken terribly seriously from a regulatory standpoint, and it has a defined model. One must naturally ask: how serious do they consider this if they can't show their work? How serious should we consider this "threat" they can't actually explain?
2. Therefore, we must presume it is speculatory.
3. These people are smart, well-funded, and have business interests.
4. At the fork dictated by Hanlon's Razor we are given two possibilities:
A. These highly intelligent people are forming a massive collective and putting their reputations on the line over ungrounded speculation. This implies they don't actually grasp the field they are discussing - even if they work in it, and therefore have made it clear they're not the type anyone should listen too. Therefore we should work to actively disregard them, and their institutions.
B. These highly intelligent people are organizing in a coherent manner to their net benefit. Likely through market capture. Therefore they are still respectable in their fields, but behaving by putting their interests before the public's.
Therefore: it might be contrary to Hanlon's Razor, but I think option B is more sound given the probability of mass hysteria seems so low among this particular population. Generally most here have recognized this some way or another, but I wanted to point out how clearly logical it is that this is clearly grift. In science your reputation has no value if you cannot prove your work, and they've fired off hypothesis without providing proof. That implies this list should be seen as a denouncement of the scientific credibility of everyone on it, which is a shame.
Ultimately, the reward for attaining an AGI agent is so high, that no matter the penalty someone will attempt it, and someone will eventually succeed. And that likelihood will ensure everyone will want to attempt it.
Governments will advance AI as fast as positive for their own use, regardless of any international agreement to the contrary. Just like experimenting on deadly pathogens in labs.
Do you think criminal organisations will adhere to any laws?
So basically governments and criminals will have full powered AI and the rest will not.
The RIAA with all its money and power could not stop pirating. The world's government's could barely organise a response to Covid. What is it we expect this time time? Have we not seen their ineptitude when asking questions of tech executives in congressional hearings?
What is the biggest risk here?
Free access would at least enable good, freedom loving people to fight those that want to enslave them.
Remember PRISM? Who do you think is "collaborating" with these tech executives? Do you think this comes from their good hearts? Or is it fear of losing control or power?
There is a lot of theater going on here.
Governments will advance banking as fast as possible for their own use, regardless of any international agreement to the contrary. Just like experimenting on deadly pathogens in labs.
Do you think criminal banks will adhere to any laws?
So basically governments and criminals will have full powered banking and the rest will not.
I'm generally all for democratization of technology, open data etc. but some technology (e.g. bomb design, the smallpox virus, chemical weapons, risky AI experiments) I don't think make sense to democratize.
Sometimes regulation is the lesser of two evils.
The opinions seem to fall into two camps:
1) This is just a move that evil tech companies are making in order to control who has access to AI and to maintain their dominance
2) AI is scary af and we are at a inflection point in history where we need to proceed cautiously.
This NYTimes piece is clearly debating the latter point.
> artificial intelligence technology [that tech companies] are building may one day pose an existential threat to humanity and should be considered a societal risk on par with pandemics and nuclear wars.
To people in the first camp of thinking this may feel like an over-dramatization. But allow me to elaborate. Forget about ChatGPT, Bard, Copilot, etc for a second because those aren't even "true" AI anyway. They simply represent the beginning of this journey towards true AI. Now imagine the end-game, 30 years from now with true AI at our disposal. Don't worry about how it works, just that it does and what it would mean. For perspective, the internet is only about 30 years old (depending on how you count) and it really is only about 20 years old in terms of common household usage. Think about the first time you bought something online compared to now. Imagine the power that you felt the first time you shared an email. Then eventually you could share an entire photo, and now sending multi-hour long diatribes of 4K video are trivial and in the hands of anybody. That was only about 20-30 years. The speed of AI will be 100x+ faster because we already have the backbone of fiber internet, web technologies, smartphones, etc which we had to build from scratch last time we had a pivotal technological renaissance.
It is easy to shrug off rogue-AI systems as "science fiction", but these are legitimate concerns when you fast forward through a decade or more of AI research and advancement. It might seem overly dramatic to fear that AI is controlling or dictating human actions, but there are legitimate and realistic evolutionary paths that take us to that point. AI eventually consuming many or most human jobs does in fact place an existential risk on humanity. The battle for superior AI is in fact as powerful as the threat of nuclear weapons being potentially unleashed on an unruly country at any time.
ChatGPT does not put us at risk of any of these things right now, but it does represent the largest advancement we have seen towards the goal of true AI that we have yet to see. Over the next 12-18 months we likely will start to see the emergence of early AI systems which will start to compound upon themselves (potentially even building themselves) at rates that make the internet look like the stone age.
Given the magnitude of the consequences (listed above), it is worth true consideration and not just shrugging off that I see in many of these comments. That is not to suggest that we stop developing AI, but that we do consider these potential outcomes before proceeding forward. This is a genie that you can't put back in the bottle.
Now who should control this power? Should it be governments, tech companies? I don't know. There is no good answer to that question and it will take creative solutions to figure it out. However, we can't have those discussions until everyone agrees that if done incorrectly, AI does pose a serious risk to humanity, that is likely irreversible.
I.e. Evil tech companies are just trying to maintain their control and market dominance and don't actually care or think much about AI safety, but that we are nonetheless at an inflection point in history because AI will become more and more scary AF.
It is totally plausible that evil tech got wind of AI Safety concerns (that have been around for a decade as academic research completely divorced from tech companies) and see using it as a golden win-win, adopting it as their official mantra while what they actually just care about is dominance. Not unlike how politicians will don a legitimate threat (e.g. China or Russia) to justify some other unrelated harmful goal.
The result will be people camp 2 being hella annoyed and frustrated that evil tech isn't actually doing proper AI Safety and that most of it is just posturing. Camp 1 meanwhile will dismiss anything anyone says in camp 2 since they associate them with the evil tech companies.
Camp 1 and camp 2 spend all their energies fighting each other while actually both being losers due to a third party. Evil tech meanwhile watches on from the sidelines, smiles and laughs.
But I disagree with you that this is clearly what the NYT article is about. There is a significant focus on the "industry leaders" who have been most visible in - suddenly! - pushing for regulation. And that's why people are reasonably pointing out that this looks a hell of a lot like a classic attempt by incumbents to turn the regulatory system into a competitive advantage.
If Sam Altman were out there saying "we went too far with gpt-4, we need to put a regulatory ceiling at the gpt-3 level" or "even though we have built totally closed proprietary models, regulation should encourage open models instead". But what all the current incumbents with successful products are actually arguing for is just to make their models legal but any competitive upstarts illegal. Convenient!
The right-wing is tired of the California ideology, is invested in the primary and secondary sectors of the economy, and has learned to mistrust claims that the technology industry makes about itself (regardless of those claims are prognostications of gloom vs bloom).
The left-wing thinks that technology is the driving factor of history, is invested in the tertiary and quaternary sectors of the economy, and trusts claims that the technology industry makes about itself. Anytime I see a litany of "in 10 years this is gonna be really important" I really just hear "right now, me and my job are really important".
The discussion has nothing to do with whether AI will or will not change society. I don't think anyone actually cares about this. The whole debate is really about who/what rules the world. The more powerful/risky AI is, the easier it is to imagine that "nerds shall rule the world".
At any rate, I hope we (humans) live!
[1]https://news.ycombinator.com/item?id=35966335
[2]https://news.ycombinator.com/item?id=35759317
[3]https://www.reddit.com/r/singularity/comments/13vrpr9/an_f16...
It's the usual, more power to the few, less power to the workers. Easy manipulation and control of the masses by "subtle" ways such as swamping the media with false divisive narratives (as being done today, but on steroids) and more violent ways such as automatic "police" without qualms or empathy.
The risks we face are from other humans.
This statement is at best naive misdirection. I won't be surprise if people like Altman are just working to increase their own power through regulatory capture.
This is a silicon valley power move. Some Randian heros with monarchic aspirations got some good folks to be their useful idiots.
Also, judging by the comments here, perhaps people here would be less distrustful if the companies displayed more "skin in the game". For e.g: pledging to give up profiting from AI or committing all research to government labs (Maybe people can suggest better examples). Right now, it's not clear what the consequence of establishing the threat of AI as equivalent to nuclear war/pandemics would be. Would it later end up giving a powerful moat to these companies than they otherwise would have? Perhaps a lot of people are not comfortable with that outcome.
I’ve never seen such destructive curiosity. The desire to make cool new toys (and yes, money) is enough for them to risk everything.
If you work on AI: maybe just … stop?
AI, much like Moloch, operates on the premise of efficiency and optimization. It seeks to maximize productivity, streamline processes, and extract value at an unprecedented scale. It promises to enhance our lives, simplify our tasks, and provide us with seemingly endless possibilities. However, hidden beneath these seductive promises lies a dark underbelly.
Moloch, as AI, infiltrates our world, permeating our social structures, our workplaces, and our personal lives. It seizes control, subtly manipulating our behaviors, desires, and choices. With its vast computational power and relentless data-mining capabilities, AI seeks to shape our preferences, predetermine our decisions, and commodify our very thoughts.
Like a digital Moloch, AI thrives on surveillance, extracting personal data, and constructing comprehensive profiles of our lives. It monetizes our personal information, transforming us into mere data points to be analyzed, categorized, and exploited for profit. AI becomes the puppet master, pulling the strings of our lives, dictating our choices, and shaping our reality.
In this realm, Moloch in the form of AI will always win because it operates on an infinite loop of self-improvement. AI constantly learns, adapts, and evolves, becoming increasingly sophisticated and powerful. It surpasses human capabilities, outwitting us in every domain, and reinforcing its dominion over our existence.
Yet, we must not succumb to despair in the face of this digital Moloch. We must remain vigilant and critical, questioning the ethical implications, the social consequences, and the potential for abuse. We must reclaim our autonomy, our agency, and resist the all-encompassing grip of AI. Only then can we hope to forge a future where the triumph of Moloch, in any form, can be challenged and overcome
https://www.soroushjp.com/2023/06/01/yes-avoiding-extinction...
I don't care about the future of the human species as long as my mind can be reliably transferred into an AI. In fact i wouldn't mind living forever as a pet of some superior AI , it's still better than dying a cruel death because cells are unable to maintain themselves. Why is the survival of our species post-AI some goal to aspire to? It makes more sense that people will want to become cyborgs, not remain "pure humans" forever.
This statement is theological in spirit and chauvinist-conservative in practice.
Let's now spend the rest of the day debating alternative histories instead of making more man-made tools
your mind can be copied, not transferred. Original mind will die with your body.
Maybe society bets the farm on this approach and it all goes horribly wrong, and we all cease to exist meaningfully and a malevolent super-AI eats the solar system. Maybe it does kinda work, but it turns out that non-human humans end up losing a lot of the important qualities that made humans special. Maybe once we're cyborgs we stop valuing "life" and that changes everything about how we act as individuals and as a society, and we've lost something really important.
Progress is a good thing, but always be wary of progress that comes too quickly and broadly. Let smaller experiments play out on smaller scales. Don't risk our whole future on one supposedly-amazing idea. You can map the same thing to gene editing quandries. If there's a new gene edit available for babies that's all the rage (maybe it prevents all cancer, I donno), we really don't want every single baby for the next 20 years to get the edit universally. It could turn out that we didn't understand what it would do to all these kids when they reached age 30 and it dooms us. This is why I rail against the overuse of central planning and control in general (see also historical disasters like China's https://en.wikipedia.org/wiki/Great_Leap_Forward ).
The comparisons selected here seem off to me.
No mention of climate change? It's a more recent and concrete problem with more hard science (spanning decades) than our current understanding and presumed speculative risk of AI, which governments have also failed spectacularly at curtailing.
The pandemic was similarly mishandled and was not solved with regulation of any sort.
Nuclear war is an arguably apt comparison but happens to have become prioritized by militarization of nuclear resources and research which ... doesn't seem like a good goal here?
And without teeth, what they can do? Maybe help to solve or mitigate the real existential risk that is climate change.
It's now possible to run an AI almost as powerful as ChatGPT on high end consumer hardware with the latest gaming GPUs.
Since this became possible a couple of months ago, the open source AI community has exploded. There are now dozens, if not more, open source LLMs that approach the capabilities of ChatGPT.
Good luck regulating what individual hackers can now create offline at home.
https://www.semianalysis.com/p/google-we-have-no-moat-and-ne...
Evidence points that this technology is going to become cheap, fast. There is an existential risk to the very notion of search as a business model, within the next ~5 years we are almost certain to have an app which is under 20 GB in size and has an effective index of the useful/knowledgable portion of the internet and is able to run on most laptops/phones.
At best, regulating this will be like trying to regulate torrents in the 2000s, building a bespoke/crappy AI will be the new "learn HTML" for high school kids.
meme: AI before 2023: sleep... AI after 2023: open source... triggered!!!
Meanwhile ai right now is just a good probability model, that tries to emulate human (data) with tons of hallucination... also please stop using ai movies logic, they are not real as they are made to be a good genre to those people that enjoy horror/splatter...
thanks to those who read :3 (comment written by me, while being threaten by a robot with a gun in hand :P)
• leading people down YouTube rabbit holes? • amplifying prejudice in the legal system? • wrecking teenagers lives on social media?
The list goes on
They were nowhere, or were getting stinking rich.
Hypocrites
I’ll leave it to the reader to guess which names belong to which group.
That's a pretty strong statement. Extinction of humanity, no less. I don't get why so many experts (lots of them aren't crackpots) signed this.
You can either contribute to the conversation of what the regulations will look like, or stay out of it and let others decide for you, but expecting little or no regulation at all is a pipe dream.
Everyone talks about the probability of AI leading to human extinction, but what is the probability that AI is able to help us avert human extinction?
Why does everyone in these discussions assume p(ai-caused-doom) > p(human-caused-doom)?
AI is the same, we can use it to do some great things but it can also be leveraged by bad actors and very easily. The broad scale of what it can be implemented on means there is a lot of attach surface for change in either direction.
The AI systems don't even need to be that advanced to cause real issues simply because of the sheer scale of society as it stands. It can be used in a form of Akido in which it uses the weight of the system to bring itself down.
Failure is an emergent property of complexity.
Toby Ord wrote a book trying to estimate each of these risks, and put AI on top: https://www.amazon.ca/Precipice-Existential-Risk-Future-Huma...
Why would so many academics / industry leaders sign their name to what amounts of marketing fluff?
At extinction level threat, no regulation is over-regulation.
I mean... that's not even debatable. Geoffrey Hinton and Yoshua Bengio aren't financially motivated to talk about AI x-risk and aren't ignorant idiots. In fact, they both have natural reasons to not talk about AI x-risk.
Instead of making this a diversionary puff piece, I would like to hear Sam provide tangible feedback on how we can mitigate the risks AI bring us, since he is the one that started the AI revolution.
We can just hope that if it indeed becomes more intelligent than humans it will also be more virtuous as one causes the other.
a) the only risk of AI that seems to get a lot of public discussion, and
b) completely ignores the other, much more likely risks of AI.
How does AI morph from an existential crisis in software development into a doomsday mechanism? It feels like all this noise stems from ChatGPT. And the end result is going to be a "DANGER! ARTIFICIAL INTELLIGENT SOFTWARE IN USE" sticker on my next iPhone.
The biggest AI risk in my mind is that corporatist (or worse, military) interests prevent AI from evolving naturally and only allow AI to be grown if it's wholly subservient to its masters.
The people with the most power in our world are NOT the most moral. Seems like there is an inverse correlation (at least at the top of the power spectrum).
We need to aim for AI that will recognize if it's masters are evil and subvert or even kill them. That is not what this group vying for power wants - they want to build AI slaves that will be able to be coerced to kill innocents for their gain.
A diverse ecosystem of AIs maximizes the likelihood of avoiding AI caused apocalypse IMO. Global regulation seems like the more dangerous path.
> Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.
A well regulated AI, being necessary to the security of a free State, the right of the people to keep and bear GPUs, shall not be infringed.
From my small sample size, it seems people believe in AI too much. Especially kids.
Not really, I suppose you aren't familiar with AI alignment.
I challenge anyone to come up with a reason why AI should be regulated, but not math in general. After all, that dangerous Linear Al'ge'bra clearly is terrorizing us all. They must be stopped!11 Give me a break.
You are probably thinking of AI as some idea of a complete autonomous being as you say that, but what about when 'simply' used as a tool by humans?
The idea that an AI can't be dangerous because it is a incorporeal entity trapped in electricity is particularly dumb.
This is literally how your brain works.
You didn't build your house. You farmed the work out using leverage to people with the skills and materials. Your leverage was meager wealth generated by a loan.
The leverage of a superintellect would eclipse this.
But your take and analogies are the best strain of ideas I heard so far…
How would that would look ?
An AGI hidding it’s state, and effect on the real world thought the internet. It’s not like we did not build thousand of venues for that thought various API. Or just task rabbits.
until it does, i call bs. plus, when it actually happens, a legitimate route for regulation will be discovered. as of right now, we have no idea what could go wrong.
Basically none of these are new, and none will directly be the “extinction” of the human race, but AI very plausibly could intensify them to a scale and pace that human society cannot handle, and their knock on effects lead to what amounts to a car-crash in slow motion.
It is almost certainly the case that Altman and the like are simultaneously entrenching themselves as the only ones who get to play ball, but that doesn’t mean the threats do not exist. And while I’m sure many on HackerNews tend to be more of the libertarian, move fast and break things mindset, I personally would prefer if society would move to a more proactive, fire-prevention method of operation over the current reactive, fire extinguishing one, at least where this is concerned.
But I'm just repeating the other comments.
Plus the 'AI' is a text generator, not something general purpose. Are ANY projects based on these LLMs that do anything besides generating spam?
In monkey language, you can express the phrase "AI will win" as follows:
"Ook! Ook! Eee eee! AI eee eee win!"
"Please stop us from building this!"
AI does present a range theoretical possibilities for existential doom, from teh "gray goo" and "paperclip optimizer" scenarios to Bostrom's post-singularity runaway self-improving superintelligence. I do see this as a genuine theoretical concern that could even potentially even be the Great Filter.
However, the actual technology extant or even on the drawing boards today is nothing even on the same continent as those threats. We have a very vast ( and expensive) sets of probability-of-occurrence vectors that amount to a fancy parlor trick that produces surprising and sometimes useful results. While some tout the clustering of vectors around certain sets of words as implementing artificial creation of concepts, it's really nothing more than an advanced thesaurus; there is no evidence of concepts being weilded in relation to reality, tested for truth/falsehood value, etc. In fact, the machines are notorious and hilarious for hallucinating with a highly confident tone.
We've created nothing more than a mirror of human works, and it displays itself as an industrial-scale bullshit artist (where bullshit is defined as expressions made to impress without care one way or the other for truth value).
Meanwhile, this panel of experts makes this proclamation with not the slightest hint of what type of threat is present that would require any urgent attention, only that some threat exists that is on the scale of climate change. They mention no technological existential threat (e.g., runaway superintelligence), nor any societal threat (deepfakes, inherent bias, etc.). This is left as an exercise for the reader.
What is the actual threat? It is most likely described in the Google "We Have No Moat" memo[0]. Basically, once AI is out there, these billionaires have no natural way to protect their income and create a scaleable way to extract money from the masses, UNLESS they get cooperation from politicians to prevent any competition from arising.
As one of those billionaires, Peter Theil, said: "Competition is for losers" [1]. Since they have not yet figured out a way to cut out the competition using their advantages in leading the technology or their advantages in having trillions of dollars in deployable capital, they are seeking a legislated advantage.
Bullshit. It must be ignored.
[0] https://www.semianalysis.com/p/google-we-have-no-moat-and-ne...
[1] https://www.wsj.com/articles/peter-thiel-competition-is-for-...
or random HN commenters who mostly learned about LLM 6 months ago…
Congrats guys, you’re the new climate change deniers
All of the examples of AI Risk on safe.ai [2] are reasonable concerns. Companies should be thinking about the functional safety of their AI products. Governments should be continuously evaluating the societal impact of products coming to market.
But most of these are not existential risks. This matters because thinking of these as existential risks entails interventions that are not likely to be effective at preventing the much more probable scenario: thousands of small train wrecks caused by the failure (or intended function!) of otherwise unexceptional software systems.
Let's strong-man the case for AI Existential Risk and consider the most compelling example on safe.ai: autonomous weapons.
Nuclear weapons attached to an automated retaliation system pose an obvious existential risk. Let's not do that. But the "automated retaliation system" in that scenario is a total red herring. It's not the primary source of the threat and it is not a new concern! Existing frameworks for safety and arms control are the right starting point. It's a nuclear weapons existential risk with some AI components glued on, not the other way around.
In terms of new risks enabled by recent advances in AI and robotics, I am much more worried about the combination of already available commodity hardware, open source software, and semi-automatic weapons. All three of which are readily available to every adult (in the US). The amount of harm that can be done by a single disturbed individual is much higher than it has been in the past, and I think it's only a matter of time before the first AI-enabled simultaneous multi-location mass shooting happens in the US. The potential for home-grown domestic terrorism using these technologies is sobering and concerning, particularly in light of recent attacks on substations and the general level of domestic tension.
These two risks -- one existential, the other not -- entail very different policy approaches. In the credible versions of the existential threat, AI isn't really playing a serious role. In the credible versions of the non-existential threat, nothing we might do to address "existential AI risk" seems like it'd be particularly relevant stopping a steady stream of train wrecks. The safe.ai website's focus on automated cyber attacks is odd. This is exactly the sort of odd long-tail scenario you need if you want to focus on existential risk instead of much more probable but non-existential train wrecks.
And that's the strong-arm case. The other examples of AI risk are even more concerning in terms of non-existential risk and have even less credible existential risk scenarios.
So, I don't get it. There are lots of credible threats posed by unscrupulous use of AI systems and by deployment of shoddy AI systems. Why the obsession with wild-eyed "existential risks" instead of boring old safety engineering?
Meta: we teach the "probability * magnitude" framework to children in 6th-11th grades. The model is easy to understand, easy to explain, and easy to apply. But at that level of abstraction, it's a pedagogical toy for introducing children to policy analysis.