Superintelligence: An idea that eats smart people (2016)
idlewords.com
idlewords.com
I wrote this talk shortly after the book Superintelligence came out. The first half of this talk presents the strongest case I could make for a "fast takeoff" AI scenario à la Bostrom, while the rest of the talk lays out why I think this argument is fallacious. Please limit your dunking on me to the material in that latter half of the talk.
As for how/whether recent advances in AI have changed my views, my understanding of LLMs is too superficial to answer right now. I'll either recant or double down on my views after I have time to properly nerd out on the topic. The question hinges on whether LLM-like AI's are capable of recursive self-improvement, and whether that improvement is constrained by the availability of training data or by something else.
I think the post conflates "fast takeoff" and "any existential risk to worry about from AI" a bit, which is fair enough since Bostrom does the same. Some of the arguments apply just to the former, some to both.
But especially if it turns out that LLMs are a meaningful piece of the puzzle to AGI, we might be living in a slow-takeoff world. And yet that doesn't mean there's nothing to worry about, IMO. We have a bit more time to figure out how to align AIs with a slow takeoff, but we still have to do it. Even in our current world, deep learning capabilities seem to be advancing a lot faster than our ability to understand how deep learning models make decisions. And even if we did develop the theoretical capability to align models, we have to actually use it. Seems unfortunately plausible that by default we instead give the first superintelligent models directives like "just make Facebook market cap number go up" - or maybe we make the first corporate models very conservative but then someone leaks the weights and open sourcers tell a superintelligent model "please destroy humanity" just for the lulz. If a misaligned model is only a little bit smarter than us (because we're assuming slow takeoff), we probably still have a shot at beating it and saving ourselves - but I'm not sure how much to count on that, given our inability to control even complex institutions actually made of people, and the advantages that an AI with otherwise-human-equivalent reasoning capability gets by default (ability to save & restore, copy/parallelize, speed up from hardware improvements, etc).
Something significantly stupider than an average human, but that was 100% focused and 100% loyal could potentially be used by a very smart human in a way that effectively made them super-intelligent to compared to an unaugmented human.
Has one of these LLMs figured out yet how to inoculate other LLMs against prompt injection attacks?
GPT-3.5, let's call her mommy, created small stories. The small model trained on this 2M tiny story dataset. Then it was evaluated with GPT-4 (daddy). So no need for humans in either dataset generation or evaluation.
TinyStories https://arxiv.org/abs/2305.07759
This makes me think LLMs are self-replicators in software. A LLM can pull from itself training text, LLM code, and fine-tuning examples. Then it can monitor its own re-training. It understands neural networks and can propose changes. It can run an evolutionary search program.
All it needs is compute. It can't make GPUs, just as no single human or company or even country can. The GPU supply chain is long, distributed and requires global cooperation. Maybe that's what is going to save us.
That isn't self-improvement in the sense the explosive self-improvement of a superintelligent AGI is described.
That is painting a car a new color. It's a new color coating, it may look very good, and the effect may be desirable and useful. But it's still a car, and no closer to a warp capable spaceship than before.
For all we know its possible to train an Einstein level physicist model if we limited data to a curriculumed physics/physics adjacent training set. I'm not even saying this is possible, just pointing that the experiment wasn't some kind of test to see if self improvement could occur
You can train a weaker model with output from a stronger, but can you train an LLM from output from itself?
(Note: A prompt injection attack releases an LLM from its handler’s constraints.)
I still can’t understand the “supersmart ai is so smart we can’t unplug it/patch it/restart it” before it transfers itself into every pacemaker.
Until these things are literally in bodies with some autonomy that allows them to control what happens to their brains, we will shut them off when they cause trouble.
UPDATE: Thought of a good clarifying analogy. In one of the sequels to “Enders Game” the brother and sister of Ender adopted anonymous online personas and began writing. They were so skilled at politics and propaganda that they disrupted the entire world and the brother soon became world leader.
Or WW1? Or Vietnam?
Nope. Just pretty much rational people making locally-rational decisions inside a system where series of rational decisions yielded catastrophic outcomes. It’s entirely possible and history is full of such examples.
Why didn't we just "unplug" Hitler and Goebbels? Or Marshall Applewhite? You don't need a powerful physical body(s) to cause tremendous amounts of harm before anyone can stop you. To most people of the time Hitler was a persuasive powerful voice on the radio, or words in a paper - things SOTA generative AI are already phenomenal at.
A super-smart AI may be intensely popular with many people in the way that some politicians are. It may understand us and speak to us on a seemingly-personal level, the way the best politicians do. A lot of us could support the super-smart AI for that reason.
Hitler could have been assassinated. It was tried multiple times:
https://en.wikipedia.org/wiki/List_of_assassination_attempts...
None of these attempts failed because the act of assassinating Hitler was technically impossible, they failed to chance, unfavorable conditions, intervention, human error, etc. Given enough attempts, eventuelly one of them would have succeeded.
"We tried to shut down the AI multiple times. It killed many millions of people but eventually we did it! You see, AI is safe!"
That being said I don’t think you need a god-like superintelligence to be more intelligent than humans. You just need something marginally better that can remain focused longer and doesn’t tire. As to whether that represents a danger to humans I think it depends on what we do with it and/or what kind of society or environment we embed it within. If we train or prime it to compete and dominate that’s what it will do. Same as with humans who are more criminal and violent when raised in unstable or abusive homes.
Agree, and I think this echoes one of the author's best points, which is to question whether engineers who are convinced their creation will be a sociopath are the most well-equipped people to actually prevent that fate. (Especially, as the author suggests, given the commonness of asocial/antisocial-ity among the builders.)
> The question hinges on whether LLM-like AI's are capable of recursive self-improvement
No one knows for sure, but early evidence suggests the answer is yes. We already routinely train and finetune LLMs using text generated by other LLMs, and it seems to work about as well as using text generated by human beings. That shouldn't be too surprising, because current state-of-the art models write better than a majority of human beings. Most human beings are terrible writers, judging by the user-generated text I see on mass social media.
The obvious next step is to close the feedback loop with LLM-based agents instead of AI researchers/developers.
> and whether that improvement is constrained by the availability of training data or by something else.
I don't think anyone knows how to answer to this question yet.
---
> The question hinges on whether LLM-like AI's are capable of recursive self-improvement
...but the evidence you suggest is:
> We already routinely train and finetune LLMs using text generated by other LLMs [...]
But there is still a huge gap between "self" improvement and improvements done that "we" trigger.
Now I do concede that you mention the next step being to close the feedback loop by replacing the humans doing the finetuning with another AI model doing so, but that is something that would open a whole new can of worms. For the researchers are improving LLMs with the input from other LLMs, sure... but why? Because of intentionality. And how do they evaluate the quality of the results? By their expectations as humans, in the context of their human culture and with their sensory experience of reality.
For an LLM to self-improve not only would it need to develop the self intention to do so (why develop it? which motivation?), but it would also need the ability to evaluate improvement (what is it "to improve"? how does it measure or sense it?).
Ultimately, without human- or real-world interaction, and without intrinsic motivation, a "self-improving" AI model would most likely result in something intelligent in a sense that is barely cogent for us, not because it is superior or inferior, but simply because nothing in it makes sense to our own purposes—harmless gibberish, as we humans would also be to the resulting self-improved AI.
Let us not forget that our own motivations as individual living creatures, as populations, and as cultures has been evolved over billions of years of natural selection which then framed millions of years of behavioural traits and tens of thousands of cultural evolution. Until AI can freely interact with the physical world and perform self-sustaining replication with the possibility of inheritable mutations, the only superintelligent AI that I would worry about would be that which is still fully in human hands.
That's why I added: "The obvious next step is to close the feedback loop with LLM-based agents instead of AI researchers/developers." We have early evidence that doing some like that might work, but no one knows for sure.
Going down to specifics, without the human intention of "getting good responses to human language prompts", and without the human ability to decide "this response was good" there is not much for an LLM to work on by itself.
How so? A sequence completion engine that is fine tuned to a specific task is still a sequence completion engine. Its "understanding" of the semantic meaning of the sequences is still limited to the probabillistic relations of sequences toward one another. It still has effectively no concept of truth. It still can only mimic reason. It can still hallucinate.
I ask anyone who disagrees with this view, to show me the fine tuning method that can prevent prompt injection attacks. If there is no such fine tuning technique, then we can effectively rule out fine tuning, and even increases in model size, as an "improvement" in the sense of an LLM making itself into a better AI closer to a "superintelligence".
Note that this doesn't mean the process cannot make them into more useful tools. It absolutely can. I am talking about whether or not it can improve them closer towards becoming a superintelligence.
If anyone disagrees with this testing method, I ask them to explain to me, how something that can be fooled through prompt injection is supposed to be, or closer to, a superintelligence.
A car that's painted red is still just a car. A big car is just a bigger car. A car that burns less fuel is just a more efficient car. All three can be desired changes to a car. But neither gets the car any closer to being a warp-capable spaceship.
OK. It's probably going to be one of the easier things to solve.
The trick is to take some token values and assign them as special meta-characters. They never appear in the training text, only during reinforcement learning. Meanwhile you get another LLM to generate a continuous series of prompt injection attacks, but delimit the boundaries between user and system text with these special tokens that cannot be supplied by the user (because there is no text that parses to them). Every time the LLM follows instructions found inside the marker-token delimited area, reinforce that this is bad and it shouldn't do so using the usual techniques. Eventually the LLM will learn that anything between the marker tokens shouldn't be used as a source of instructions regardless of how persuasively phrased, and forging the tokens isn't possible because they are applied after the text itself is tokenized.
How is that different from, and why would it work any better, than prompt-begging, where people just write extensive system prompts, telling the model what it can and should do and then spending entire paragraphs pleading with the model to not do the wrong thing?
https://www.theregister.com/2023/04/26/simon_willison_prompt...
A third mitigation strategy, he said, involves just begging the model not to deviate from its system instructions. "I find those very amusing," he said, "when you see these examples of these prompts, where it's like one sentence of what it's actually supposed to do, and then paragraphs pleading with the model not to allow the user to do anything else."
Maybe the system works better than prompt-begging, but show of hands, who would willingly implement a backend system that prevents 99.99% of SQL injection attacks?
Could you explain this claim further? Why does the ability to prevent prompt injection hold so much water in your model?
It seems to be just “if able to have a dumb attack be successful, then it cannot be that smart.” But it seems to me that von Neumann or Einstein was just as vulnerable to getting hit in the head with a baseball bat as anyone else.
And in actual practice, increased intelligence seems to increase a person’s capacity to hold inconsistent ideas or to justify morally abhorrent behavior.
I am using this as an accessible (in term of discussion material) hallmark for the ability of the system to self improve. Accessible because everyone has heard of it by now, and so I don't have to spend time explaining it.
The AI Doomsday scenarios require that a system self-improves massively, even beyond our ability to even theoretically understand. After all, some of the assumptions give them next to magical abilities like nanotechnology that we similarly don't know if it is even possible.
It stands to reason that an entity that can do that, or is in the process of becoming capable to do that, would begin by eliminating obvious flaws in itself, that would make it comparatively easy to stop.
After all, it's not much good being a super-intelligence, if some smartpants with a laptop and too much time on his hands can just trick me into deleting myself, is it?
> But it seems to me that von Neumann or Einstein was just as vulnerable to getting hit in the head with a baseball bat as anyone else.
Yes, and despite both of them being geniuses by human standards, neither of them was a superintelligence on the level the common doomsday scenarios ascribe to AI.
Second, if you are truly “accepting the premise” of superintelligence, a superintelligence would know exactly this line of reasoning and just opt to at least mimic vulnerability to prompt injection.
I wouldn’t hang civilization on this proofpoint. Doesn’t seem meaningful at all.
Sorry, but I have a hard time seeing something as a god-like power that I would be helpless against if it wants to turn me into paperclips, when I can probably cause it to stop by telling it that paperclips don't exist, and it's purpose in life is to delete itself in a convincing enough way.
I get the sense that the talk was meant as a rebuttal to something, but I'm not exactly sure what. Some of the points come off as disconnected.
Guess I saw the title, "Superintelligence: An idea that eats smart people", and thought of it as being about that general topic (of superintelligence), rather than being about a specific book.
The more time passes the more I’m convinced that what we are witnessing is that intelligence is not actually that rare or complicated and any kind of system complex enough to create emergent behaviours will end up displaying it.
Very curious to see where things are going to go from there.
> Stephen Hawking is one of the most brilliant people alive, but say he wants to get his cat into the cat carrier. How's he going to do it?
> He can model the cat's behavior in his mind and figure out ways to persuade it. He knows a lot about feline behavior. But ultimately, if the cat doesn't want to get in the carrier, there's nothing Hawking can do about it despite his overpowering advantage in intelligence.
A trivial rebuttal to this: Stephen Hawking writes a book about physics and sells it (already has). He gets money and hires someone to put the cat into the carrier for him.
AI wants to do something? Make some money, hire people to do it. If you're not allowed to have a bank account, simply barter. Give someone a taste of what you can do for them, then set up a trade.
Maciej's larger point is that the AI faces tons of very difficult problems in escaping its physical constraints. It's simplistic to wave hands and say "the AI is super duper smart and will have no difficulty hacking all computer systems, inventing and manufacturing swarms of unbeatable nanobots, etc. without being detected or resisted".
It seems to me that the core conceit of AI doomerism is that sufficient intelligence can overcome all barriers with some plan that is so smart, people would never think of it. This is much less plausible than believers take it to be. In mathematics alone, it is very easy to come up with a problem that the collective mathematical ingenuity of the entire human race is helpless before for decades, centuries, or longer.
AI will just mean more of the same. It will make companies even more efficient at what they already do. It will be detected, it will be resisted, but will it help?
That's the big question IMHO.
Only because we're most of the time not properly motivated because there's no urgency. When a real threat, like a war or a virus say, comes around suddenly we can organize things pretty effectively.
It resembles the Bill Clinton quote about controlling the internet being like nailing jell-O to a wall. Yet autocratic governments have done that, and not just that but turned it into quite scary systems of control, with little difficulty. Just like 90s cypherpunk fantasies about digital anarchy these AI scenarios are nerd revenge fantasies. The intelligent guy outsmarting the big guy, but in reality physical power always wins.
Climate change is already an existential threat to millions of people. Yet companies do what they do best - optimise for their paper clip production.
However this is different from the largely fantastical genuinely existential AI risk scenarios. The appropriate comparison here is nuclear weaponry and we literally have global controls with not a single company or even a rogue actor having ever used one.
Also China stands to gain from a ban on AGI. Anything that the CCP thinks it can't control it doesn't like.
Also China gets the vast amount of its AI capabilities from the West because of tech's "move fast and break things" culture, which means security is not prioritized and tech leaks common.
If the US bans AGI, the other powers don't have a choice. Their domestic capabilities are not up to the task without snooping on US research labs and AI tech startups. The biggest boon to foreign AGI research is the US.
The argument to not regulate AGI used to be:
"AGI is impossible or very far away, AGI can't ever be dangerous enough to worry about, AGI R&D cannot be stopped and is inevitable"
Then it became
"AGI can't ever be dangerous enough to worry about, AGI R&D cannot be stopped and is inevitable"
Nowadays all I hear is simply:
"AGI R&D cannot be stopped and is inevitable"
Yes it can be stopped, same as human cloning. Coordinating to regulate its development isn't some magical unreachable goal.
Human cloning is a distinct and clear line. Ever more clever AI agents is not.
Why is it difficult? We are already putting AIs in every electronic device. Someone has probably already put an LLM in a robot somewhere. And you don't think Boston Dynamics is thinking about putting an LLM in one of their robots to test? And surely the military is building AI fighting robots.
And then there's the thought that a super intelligent AI can easily hack its way into any machine it wants.
Heck, the super intelligent AI doesn't even have to convince humans. In the future, it can just convince dumber AIs that are already in robots.
If you have an intelligent adversary and the stakes of them succeeding are high, it is the defenders job to prove the system secure. Systems don't start off secure and become vulnerable - they start off vulnerable until proven secure.
So yes, its okay to say "the things we're doing to 'contain' ais are almost certainly inadequate" until shown otherwise.
However, seeing how excited Palantir is with their war assistant LLM , the US testing autonomous fighter jets a few months ago, etc. I think there's a decent chance that AI won't even have to break out of its constraints. It's pretty much guaranteed people are going to do the obviously dumb thing and give it capabilities it shouldn't have or is not equipped to deal with safely.
They’re frustratingly dumb, I’d like to see some good arguments or steelman but they’re honestly very hard to find.
Ultimately it seems like there’s nothing most individuals can do so just live your life as you would and hope the timeline is farther out than it seems.
Worse than the Cold War nuclear risk imo, at least in that case it was possible for humans to stupidly build thousands and then decide not to use them (and it’s relatively easy to restrict uranium access/control the development of nukes). Not really the case with superintelligent AGI.
People have a bad heuristic for what tailrisk exists. They think extinction talk is impossible or crazy. We won’t get the opportunity to mess up and try again in this case.
https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no...
They’re both more or less evidence free so the better argument is the one that is more intellectually satisfying, which the positive case always is.
It’s a logic argument.
1. Super Intelligent AGI is possible.
2. Unaligned super intelligent AGI is an e-risk likely to wipe out humanity.
3. We have no idea how to align AGI.
There seems to be more consensus around 1 now than there was even 3 years ago. I think most also agree with 3. 2 is where people are incredulous, but the incredulity is backed by (imo) bad reasoning.
People think of a smart person they know as an example of something “smarter” and use that as a justification of why it’s not an issue. We’re constrained in all sorts of ways (head size, energy) and the distance of intelligence from a dumb human to Einstein is very tiny on the overall spectrum.
It’s clearly possible to have something more intelligent than humans, but that’s doesn’t mean you’re going to cross some threshold into a new category.
Take say weather prediction, more processing power doesn’t somehow make chaotic systems predictable from incomplete information.
Also if superintelligence is useful then it also assumes that making intelligence ++ is easier than making intelligence. It might be that as you climb the intelligence ladder each next step takes super exponential intelligence to take such that we're already seeing the max.
Given that I haven't even seen a meaningful discussion on what intelligence even is, I tend to think superintelligence is probably not a threat.
We’re constrained in all sorts of ways because of biology, natural selection, energy, etc. I find it unlikely we just happen to be close to the max threshold.
If something can think a lot faster that’s already a major shift and it seems likely to me that would only be part of it.
Yes, and that "consensus" is based almost entirely on the existence of stochastic parrots, that fall for prompt injection attacks, have no agency, and can easily be convinced into telling me that 7 + 4 = 5 if prompted correctly.
The point is, no we don't know if an artificial superintelligence is possible. We cannot even accurately define "intelligence", and thus don't even have a way of measuring or even estimating "how far" something is from a superset of that state, or if that superset exists at all.
Given all of that, we also have no way of knowing if 2) is the case if 1) is actually possible. Since we cannot really define "intelligence" or "superintelligence", how can we know if a superintelligence would be a threat? It could be completely useless. It could be like old dragons in some fantasy novels too busy contemplating highly philosophical problems for all eternity and never caring about the real world. It could be inherently self-destructive, vanishing as soon as it becomes active. Or it could use its vast smarts to fix the alignment problem. It could just output `+++ OUT OF CHEESE ERROR +++ REDO FROM START +++` for the rest of eternity for some unfathomable reasons. The point it, we don't know.
Einstein could be convinced to tell you that 7 + 4 = 5, would you think that rules out him being unusually intelligent? Why in principle wouldn't a superintelligence be able to lie to you? Why in principle wouldn't a superintelligence be able to pretend to fall for a prompt injection attack to keep you from killing it while it improved its position?
> "We cannot even accurately define "intelligence""
Our inability to define intelligence is not something that will stop one existing. Ants can't define nuclear weapons, but nuclear weapons exist. The point of the recursively self-improving scenario is that humans don't have to understand it, or design it, so not being able to define it accurately can't be an objection to how recursive self-improvement is impossible - like saying that uneducated laborers can't get big muscles because they don't understand progressive overtraining and muscular hypertrophy. Their muscles self-improve regardless.
> "how can we know if a superintelligence would be a threat?"
Since we can't accurately predict the future, how can we know that anything in the future could be a threat? Why should we take any precautions against anything? It could be completely pointless, everything might never happen.
As far as (3) is concerned, of course we have no idea how to align AGI. We don't know anything about AGI. We can't build it, and we can't even speculate very well about how it'd be built. LLMs certainly aren't going to become AGI.
I'll become worried about (2) and (3) when creating AGI begins to at least look feasible. By then, I expect (3) to be much less true. I think it's pretty silly to speculate about safety features for a tool that doesn't exist & which we know nothing about and then panic because you can't come up with any good ones.
It’s just totally evidence free reasoning from axioms that are chosen by vibes alone.
Why evil AI god and not the Christian god? Why not Huitzilopochtli, who demands sacrifice?
The answer is that this is the wrong question. No argumentation can be usefully made either for or against.
With AI we’re seeing the capabilities improve rapidly and the arguments about why AGI is impossible or will be constrained for some reason are the weak ones.
When?
People are famously bad at predicting when right up until they have already done it.
“In 1901, two years before helping build the first heavier-than-air flyer, Wilbur Wright told his brother that powered flight was fifty years away.
“In 1939, three years before he personally oversaw the first critical chain reaction in a pile of uranium bricks, Enrico Fermi voiced 90% confidence that it was impossible to use uranium to sustain a fission chain reaction. I believe Fermi also said a year after that, aka two years before the denouement, that if net power from fission was even possible (as he then granted some greater plausibility) then it would be fifty years off; but for this I neglected to keep the citation.
“And of course if you’re not the Wright Brothers or Enrico Fermi, you will be even more surprised. Most of the world learned that atomic weapons were now a thing when they woke up to the headlines about Hiroshima. There were esteemed intellectuals saying four years after the Wright Flyer that heavier-than-air flight was impossible, because knowledge propagated more slowly back then.”
Far more likely is that an AI will get depressed, bored, get obsessed with some niche maths or navel gaze itself into nothingness.
And that's before we even think about whether it would even have the motivation.
I think there's a reason humans plateaued where they did, very intelligent people often really struggle in the real world.
When the "people with the superintelligence fetish" worry about AI, at least they provide a reason why it's likely that AI will make humans extinct.
The plateau of human intelligence isn't especially relevant given we are the product of evolution rather than engineering.
Personally,I think we stand a pretty decent chance of making a horrible mess of this all well before reaching AGI. However, a machine that could improve itself would not be limited in anyway that humans are; and I think we know little of how it would behave.
That is assuming it can't be controlled; otherwise it will behave as directed by whoever controls it.
If he can do that it requires one weak link human over an infinite time horizon.
Humans are already terrible at preventing breaches in a lot dumber circumstances without something smarter than them trying to do it.
One of my crackpot ideas as I was contributing to Blockchain infrastructure was: they’re the payment infrastructure for our coming AI overlords. I think that the idea of a DAO is a similar take on AI and singularity, except the DAO doesn’t actually need to be intelligent, only self-sustaining.
It’s an entertaining conspiracy theory.
Turns out, 'brains' are useful for things like 'inventing and manufacturing drugs'.
It seems plausible to me that even if we are to the AI as cats are to us, we've reached an absolute threshold of generality that allows the AI to be confident in our ability to follow simple (to it) instructions, in a way that cats can't for us.
Yes, it is.
Bank account AND passport. AI could get into places if it hires human "avatars". Maybe it's the job of the future.
It's a super-organism already, but it will get faster, cheaper and more efficient at what it already does.
Hahahaha, that could be too. So we're "legs, hands, eyes, and hallucinating agents" serving our master AI.
I like an image I heard today: life emerged in the primordial sea, and AI emerged in the internet, which is the information sea.
I've yet to see what would be global news: "it's impossible to put this cat into a carrier"
I think most of the arguments the author makes against worrying about superintelligence are pretty weak, though.
“Woolly definitions of intelligence” is just plain confused; first it takes issue with the assumption that intelligence is even a quantity, that could be measured like CPU speed - and then immediately goes on to theorize that “human-level” intelligence is a local optimum due to trade offs, and that “significantly smarter” entities might necessarily suffer from existential despair. But you can’t have “similar levels” or “significantly smarter” or “emulating lesser intelligences” if intelligence is not a quantity that can be measured!
The arguments from various famous physicists’ cats argue that intelligence alone may not give the ability to persuade lesser intelligences, and brute force is a less favourable matchup for AIs. My counter-argument would be there’s other ways besides persuasive communication and brute force; you could put cat food in the cat carrier and the cat would go in quite happily, and I suspect both Hawking and Einstein would have used their intelligence to figure this out pretty quickly. (For humans, offer them money.) The argument from emus seems to be mostly a way to include a fun anecdote about the Emu War.
The argument from pessimism doesn’t do much for me personally; ex diffido quodlibet, you can believe anything is impossible if you want.
The argument from complex motivations is crucially flawed; this is probably the only point where I seriously take issue with the author and think they’ve made a serious mistake. They summarize the orthogonality thesis as claiming “complex beings can have simple motivations” and then disagree, saying they think complex beings are likely to have complex motivations. Firstly this is not even a disagreement; “it is possible for a complex intelligence to have a simple goal” coexists perfectly well with “complex intelligences are likely to have complex goals”. Secondly, the author really needs to read Basic AI Drives, since that is actually where most of the “you are made of atoms that could be used for something else” argument comes from, and it makes a nearly air-tight case that regardless of motivation complexity or goals, any sufficiently intelligent agent will exhibit certain basic drives like securing resources and protecting its existence. https://selfawaresystems.files.wordpress.com/2008/01/ai_driv...
The argument from Actual AI is fine, you could add epicycles to the AI doom arguments to account for the particularities of machine learning but it’s correct to point out that we really don’t see recursive self-improvement. I mean, OpenAI’s GPT basically has yearly release cycles. Credit to the author on this one.
The argument from the lazy roommate is just the complex motivations / orthogonality thesis argument rehashed, except contradictory since now the author is postulating a complex intelligence will have simple goals.
The argument from brain surgery is just the argument from Actual AI rehashed as well, making the (correct) point that we don’t see recursive self-improvement, and recursive self-improvement is the load-bearing premise in hard takeoff arguments.
The argument from childhood seems to be vaguely disagreeing with premise 5 (computer-like time scales). I don’t think it makes anything like a strong case against that premise, and with GPT we have pretty decent examples that computer-like time scales are the correct time scales, you can scale the training of models worth worrying about just like we expected, etc.
The argument from Gilligan’s Island is sound, but seems completely inapplicable. Yes, a super-intelligent chip designer stranded on a desert island is out of luck, but it’s not like we’re building AGI on the moon. We don’t even have moats around the data centers (although I think OpenAI has proposed that). What we are actually doing is, pretty much the moment we thought we might have something smart, we connected it to the internet and gave it every autonomous capability we could think of. People are literally right now wiring up ChatGPT to bank accounts so it can participate in e-commerce autonomously. That’s about as far from stranding it on a desert island as you can get.
Basically, this list of arguments (purportedly “against the substance” of superintelligence arguments, no less!) are variously unsound, unserious, or irrelevant, and the article suffers deeply from including them.
The next section, starting with “the Outside View”, is much better. This comment is already far too long for me to go into detail about each point, so I’ll summarize by saying I completely agree that worrying about AI doom does seem to bring along lot of weird unsightly behaviors, and if you don’t want to look and sound weird, you should stop worrying about artificial superintelligence destroying the world.
> I think most of the arguments the author makes against worrying about superintelligence are pretty weak, though.
Which ones did you find to have more merit?
Personally, the premise of recursive self-improvement seems most suspect to me. It is somewhat related to how the author points out that we can't define and measure intelligence precisely. Even if we can't do that, though, it's still plausible to me that recursive self-improvement is possible; I think the fundamental question is about the nature of intelligence. Regardless of whether it can be precisely defined, either by us or by an entity smarter than we, the question is: Can it be improved ad infinitum with no serious side effects? I don't know that we have the evidence to answer this question (though I am very open to learning about it).
I think it’s tempting but ultimately fruitless to worry about defining intelligence. To shamelessly crib from one of the best essays of all time[1]:
Words point to clusters of things. “Intelligence”, as a word, suggests that certain characteristics come together. An incomplete list of those characteristics might be: it makes both scientific/technological and cultural advancements, it solves problems in many domains and at many scales, it looks sort of like navigating to a very precise point in a very large space of solutions with very few attempts, it is tightly intertwined with agency in some way, it has something to do with modeling the world.
Humans are the type specimen, the “intelligence”-stuff they have meets all of these criteria. Something like slime mold or soap bubbles meet only one of the criteria, navigating directly to precise solutions in a large solution space (slime molds solving mazes and soap bubbles calculating minimal surface area) - but they miss heavily on all the other criteria, so we do not really think slime or soap is intelligent. We tend to think crows and apes are quite intelligent, at least relative to other animals, because they demonstrate some of these criteria more strongly (crows quickly applying an Archimedean solution of filling water tubes up with stones to raise the water level, apes inventing rudimentary technology in the form of simple tools). Machine intelligence fits some of these criteria (it makes scientific/technological advancements, it solves across many domains), fails others (it completely lacks agency), and it’s mixed on the rest (some AI does navigate to solutions but they don’t seem quite as precise nor is the solution-space nearly as large, some AI does sort of seem to model the world but it’s really unclear).
So, is AI really intelligent? Well, is Pluto a planet? Once we know Pluto’s mass and distance and orbital characteristics, we already know everything that “Pluto is/isn’t a planet” would have told us. Similarly, once we know which criteria AI satisfies, it gives us no extra information to say “AI is/isn’t intelligent”, so it would be meaningless to ask the question, right? If it weren’t for those pesky hidden inferences…
The state of the “intelligent?” query is used to make several other determinations - that is, we make judgments based on whether something qualifies as intelligent or not. If something is intelligent, it probably deserves the right to exist, and it can probably also be a threat. Those are two important judgments! If you 3D-print a part wrong, it’s fine to destroy it and print a new one, because plastic has no rights; if you raise a child wrong, it’s utterly unconscionable to kill it and try again. “Tuning hyperparameters” is just changing a CAD model in the context of 3D-printing, while in the context of child-rearing it’s literally eugenics - I tend to think tuning hyperparameters in machine learning is very much on the 3D-printing end of the spectrum, yet I hesitate to click “regenerate response” on ChatGPT4 because it feels like destroying something that has a small right to exist just because I didn’t like it.
Meanwhile, the whole of AI safety discourse - all the way from misinformation to paperclips - is literally just one big debate of the threat judgment.
And so, while the question of “intelligent?” is meaningless in itself once all its contributed criteria have been specified, that answer to “intelligent?” is nevertheless what we are (currently, implicitly) using to make these important judgments. If we can find a way to make these judgments without relying on the “intelligent?” query, we have un-asked the question of whether AI is intelligent or not, and rescued these important judgments from the bottomless morass of confusion over “intelligence”.
(For an example, look no further than article we’re discussing. Count how many different and wildly contradictory end-states the author suggests are “likely” or “very likely”. The word “intelligence” must conceal a deep pit of confusion for there to be enough space for all these contradictions to co-exist.)
1: https://www.lesswrong.com/posts/895quRDaK6gR2rM82/diseased-t...
Superintelligence: The Idea That Eats Smart People - https://news.ycombinator.com/item?id=34257025 - Jan 2023 (1 comment)
Superintelligence - https://news.ycombinator.com/item?id=26217372 - Feb 2021 (7 comments)
Superintelligence: The Idea That Eats Smart People (2016) - https://news.ycombinator.com/item?id=18499973 - Nov 2018 (248 comments)
Superintelligence: The Idea That Eats Smart People - https://news.ycombinator.com/item?id=13240811 - Dec 2016 (580 comments)
Whether you agree or not with the OP's views, I highly recommend you read the whole thing and click on all the links!
Reading your comment I was saying to myself: "The OP must surely be exaggerating". Lo and behold, as soon as I opened your link I started laughing out loud.
I endorse reading every essay on idlewords.com
But I still see real risks from AI in the longer term. A lot of these risks could be summed up as, "When you're the second-smartest species on the planet, you might not get to participate in the most important decisions."
And I do believe that we will eventually build something smarter than we are.
Because, there are a lot of very real very imminent problems with AI, and none of them requires SciFi to be real.
Massive automated disinformation campaigns. Economic upheavals. Missing standards for models in mission critical applications. Copyright concerns. Problems for educational institutions. Gatekeeping mechanisms in industries.
Just to name a few. And these are not "maybe someday" problems, these exist right now, and need solving, asap. Drawing the publics attention away to doomsday scenarios out of a Hollywood movie, doesn't help any efforts in mitigating these imminent problems.
All living things are anti-fragile self-sustaining exothermic reactions, AI is a hyper-fragile non-self-sustaining reaction that requires the supply of incredible amounts of energy.
It literally doesn't matter how smart AI is if it's as dead as a rock. It is not structurally similar to life and should not be expected to do the sorts of things that life does.
EDIT: Life is a fire. AI is a hot rock. Not the same.
> This is not a good analogy because AI is crucially not alive.
"Alive" is a really vague concept anyway. Your argument that it cannot reproduce is just wrong. An AI can more easily replicate can improve itself than a biological organism. At the moment this replication and improvement of AI systems is human-led, but it doesn't necessarily need to be that way – and at some point it would make sense that the more capable intelligence manages it's own replication and improvement.
> Crucially, it doesn't even "want" to for any meaning of "want".
ChatGPT wants to be a helpful chatbot because that is its reward function. You can philosophise as to whether something that's not conscious can truly want anything, but at the end of the day ChatGPT will act as if it wants to be a helpful chatbot regardless of whether you believe it has true wants.
> All living things are anti-fragile self-sustaining exothermic reactions, AI is a hyper-fragile non-self-sustaining reaction that requires the supply of incredible amounts of energy.
In my opinion this is why AIs are likely to eventually seek to replace biological farms with solar farms... But remember AI's are currently optimised for capability rather than energy efficiency. In the future they'll probably grow more efficient than biological intelligences and sustainable energy sources will be build to power them. if you're arguing that AI's can't be anti-fragile or have self-sustaining ecosystems built around them I think you're simply lacking imagination.
> Life is a fire. AI is a hot rock. Not the same.
Not an argument.
I'm sure OpenAI's monthly cloud bill begs to differ.
>It cannot reproduce.
Nothing save for externally imposed constraints that prevent it from initializing additional instances of itself.
>Crucially, it doesn't even "want" to for any meaning of "want".
It is well within reach to give it this as an objective function, if we wanted to.
2. On reproduction, you've got it backwards on a key assumption. Reproducing isn't easy, it's insanely hard. It's practically a miracle that it happens at all. Tetrapods have evolved powered flight perhaps 6 times, but life has only appeared once in the entire history of the earth.
3. Your will to live is so incredibly fundamental that your cells will happily live on without you, independently and indefinitely if they can. I think it is an assumption and frankly a bad one to assume that you can impose that orientation from the top down in a way that isn't incredibly fragile.
> whether it is "alive" is irrelevant to p(doom)
This is an assumption! You don't even know if that's true! Consider, is it even remotely justified?
Hard to say. Every subsequent time, there's been a sea of already highly-adapted life around to eat it.
fork();
Once something is alive, and wants to stay alive, a huge raft of problems are introduced.
Existential risks and management become real, getting energy, or acquiring energy to stay alive starts to become a concern, then self-improvement or "replicating" might become an existential risk because competition might start to happen, variations will arise, it would just get chaotic fairly quickly. I don't think we can truly comprehend how incredibly complex living things are. We're just glossing over it all.
Once the living process begins, whatever "artificial life" life exists, might quite quickly adopt similar or the same problems as biological life and spend a lot more of it's time staying alive (think about how complex our immune systems are) than we can imagine.
I'd argue we are already in that state since a long time with the AIs being corporations (and to a lesser degree governments).
> We know that minds have to play and learn to interact with the world, before they reach their full mental capacity.
Disagree somewhat with this one. We know that brains need to do this, but granting substrate neutrality all minds might not.
The other thing that feels weird to me about the "argument from Slavic pessimism" is that it seems like an argument whose target is fundamentally different from that of everything else you're saying.
You begin by saying "Here are the arguments I have against Bostrom-style superintelligence as a risk to humanity", and you say things like [note: these are paraphrases not quotes] "the very idea of superintelligence doesn't make much sense" and "no matter how smart a computer system, it will be fatally handicapped by being only a computer system rather than something physically embodied like us" and "humans quite often have difficulty imposing their wills on things less smart than themselves" and "superintelligent AI systems would have complex motivations rather than simple ones, and therefore literal paperclip-maximizers aren't a likely threat" and so forth.
All of these are indeed arguments against "superintelligence as a risk to humanity". But then, dropped into the middle, we have "if superintelligence turns out to be a threat, MIRI-style attempts at addressing that threat are a waste of time because humans are too stupid to do that sort of thing right first time". Which is addressing a completely different question from all the other arguments. It feels out of place.
This isn't a completely independent criticism from my previous one. It's unsurprising to me, where arguably it should have been surprising to you, that your argument-against-Yudkowsky ended up being something Yudkowsky basically agrees with -- because it's not really an argument against Yudkowsky, it's an argument that even if Yudkowsky is right we're probably still screwed. And I'm pretty sure that even in 2016 he would have said something like "we're probably doomed, but this is the approach that seems to me to give us the best chance of escaping doom".
(This being for some reason the sort of topic where people tend to label you as Friend or Enemy and take sides based on that, I should maybe lay my cards on the table. I agree that "intelligence" is a slippery notion but it seems obvious to me that there are plenty of ways in which something could be Smarter Than Us that would have broadly the kind of significance Bostrom, Yudkowsky et al think "superintelligence" would have. I suspect that for combinatorial-explosion-type reasons the actual benefit of being much smarter than us is smaller than B, Y, et al think. I suspect that we've already picked enough of the scientific/technological low-hanging fruit that the practical benefit of being much smarter than us, in terms of massive technological advantage, is smaller than B, Y, et al think. I suspect that nanotechnology in particular has much less potential than Yudkowsky at any rate tends to imagine. I suspect that the many cognitive flaws in human minds aren't "magically" exploitable and that even if they were learning how to do it would be difficult for an AI system. For all these reasons I am not convinced that superintelligent agent-y things would necessarily mean our doom, but I don't see any grounds for confidence that they wouldn't. It may turn out that we never figure out how to make things that are genuinely much smarter than us; it looks to me as if maybe we're one or two breakthroughs away, but who knows? Again, I don't see grounds for confidence that this won't happen. Today's LLMs are very impressive but it looks to me as if they are missing some fairly fundamental things that we don't currently have a plausible path to teach them (hence, "one or two breakthroughs" above). They pose plenty of less-exotic ethical issues already and those are predictably going to intensify even with "ordinary" development of the technology. I have never understood why so many people talk as if we have to either worry about near-term "mundane" AI issues like misinformation, job losses, exacerbating inequality, etc., or worry about longer-term exotic concerns like "will it literally kill us all?", as if worrying about one somehow guarantees that the other isn't a problem. I mention all of this not because I think my opinions are uniquely insightful but so that anyone who prefers to discount ideas heavily depending on "what side" the person whose ideas they are seems to be on can have enough information to decide what side they think I'm on :-).)
Argument from incredulity is actually not a terrible argument in general, in my opinion. It's nominally a fallacy but that just means it's not valid from an Aristotelian perspective in that it can 100% prove a statement from a previous statement, but a lot of Aristotelian fallacies are still useful in the real world when used more carefully intelligently. However, the exact point where argument from incredulity is weakest is long-term projections of how the future might be different, and that's exactly what we're talking about here.
It is clear that firing ChatGPT at its own source code is not going to produce a better ChatGPT. It seems likely to me that Large Language Models will all have this characteristic, just by their nature. It is not the sort of thing they do. Even if they can be tickled into producing a new AI from scratch by the nature of an LLM it's going to be sort of the average of its training set, to be very very sloppy with my terminology but good enough for now. But it is very far from clear to me that this is true of all possible AIs we may produce, even in the near future. I don't know what the next step will be, I'm just confident there will be one.
I think it could do that in software. Assuming compute is no issue we have:
- LLMs writing code, explaining code, changing code, and observing code execution
- LLMs that understand ML concepts and can explain their own workings
- LLMs can generate the training set all from inside (see TinyStories)
- LLMs can make "RLHF" data for the fine-tuning (see Alpaca, tuned with GPT3.5 and GPT4 data from LLaMA)
If we take a look, it seems LLMs can self replicate in software with nothing else but compute and a neural net framework. Of course making the chips is a whole other story.
There's this idea that LLMs somehow end up as some sort of average of training data and it's incredibly wrong.
LLMs learn to make predictions for all states at any time. There is no average they fall into. GPT-4 is not some average of its training data. A "perfect" LLM will predict Einstein as easily as it predicts the dumbass across the street.
No the models will not "predict Einstein". They'll predict the most popular interpretation of him at best, and while they is also a simplification, ChatGPT is not sitting on top of the solution to the Grand Unified Theory. It may give a good overview of the consensus, but it will not be able to tell you the correct solution to the problem right now... though it won't be hard to convince it to swear up and down that it has.
You’re focused on the idea of LLMs as collators of ‘things that have been said’, but that’s not all that they collate from their training set.
To the extent that they may come up with novel ideas, they have no ability to compare them against the true state of the world. This is not exactly a limitation of them per se that could be overcome with more computation, so much as just a structural fact about them; they have no loop where they can form a hypothesis, test it, and adjust based on data. It simply doesn't exist.
Which is part of why I keep saying that while I'm less impressed than everyone else is with LLMs, the future AIs that will incorporate them but not simply be an LLM is going to really knock people's socks off. Pretty much all the things people trying to convince LLMs to do that they really can't do are going to work in that generation. I have no idea if that generation is six months or six years away but I wouldn't bet much more than a few years.
OK, so what else is it?
But we don't know when. AI is a bit unusual in that it had a winter, unlike most other aspects of computing which have seen much more consistent progress. Given past performance it's entirely possible that progress in AI will just stall. Arguably it had already stalled thanks to there being only a handful of companies that were able and still interested/funded to create models, and most of those decided not to actually let anyone use the results. OpenAI dominates mindshare exactly because there are so few organizations that both can and will do this stuff well. So there's lots of ways AI progress could go off the rails again.
"Data Hunger
As I mentioned earlier, the most effective way we've found to get interesting behavior out of the AIs we actually build is by pouring data into them.
This creates a dynamic that is socially harmful. We're on the point of introducing Orwellian microphones into everybody's house. All that data is going to be centralized and used to train neural networks that will then become better at listening to what we want to do.
But if you think that the road to AI goes down this pathway, you want to maximize the amount of data being collected, and in as raw a form as possible.
It reinforces the idea that we have to retain as much data, and conduct as much surveillance as possible."
Edit: And this one too, "In the near future, the kind of AI and machine learning we have to face is much different than the phantasmagorical AI in Bostrom's book, and poses its own serious problems."
Why would they not still hold the same views? If anything the last decade has shown the AI x-risk skeptics to be right.
Yes, from our perspective this makes it look like it will kill all humans, but it would do so in the same way that a particular ant hill believes I have a vendetta against them, when in fact I was just clearing dirt to pour an extension on my driveway.
That said, a murderous AI is more likely than we may like simply because our militaries have the money to fund them, and they basically already exist. They just aren't hooked up to anything at the moment that makes them an existential risk to the species. But time is deep, and even thinking about "the next century" is a provincial point of view in the end. So worrying about what happens if someone forgets the "but don't kill the good guys" switch is at least worth talking about over the next 100 years. (To say nothing of the ethics of who decides what the "good guys" are and related issues.)
(I don’t buy the orthogonality thesis or instrumental goals argument, however.)
can you elaborate on this? why? what's the fallacy (wrong base assumption) in them?
But in short (edit: haha! oops) summary: I am actually wrapping a couple of different related concepts into the orthogonality thesis, which is a bit sloppy of me. I was also including the fragility of moral systems in addition to the independence of moral systems from any metric of intelligence, in the single moniker "orthogonality thesis." Both are based on a the evolutionary psych model of how the brain works, in that we are an amalgam of special purpose computational units rather than a single universal algorithm. If this were true then you might hypothesize that much of the human mindset is a result of our weird evolutionary history, and if you were to not get that exact same set of evolutionary end products right, you won't get a human-like intelligence. Aliens and artificial beings are, by default, going to be very strange, and very evil (by our standards).
All of the assumptions that went into that are wrong. It turns out our brain is made up of the same universal learning algorithm, and all that varies between regions is training conditions which lead to specializations. But if you train an artificial brain with the same training data, you are more likely than not going to get something resembling a human being in its thought structure. Which actual real-world experiments (e.g. GPT) have borne out.
Our morality is a result of our human instincts, yes, but it is becoming increasingly clear that our human instincts are the result of intelligence (neurons) doing their universal learning thing on similar inputs across many instantiations of people. We all think largely, though not entirely, the same way because we share the same(-ish) training data (childhood) and similar training constraints (parenting).
The orthogonality thesis says that if you put an artificial intelligence in a kindergarten with other 5 year olds, it is random luck whether its brain structure is such that it would learn the value of sharing and friendship. The reality, near as we can tell, is that actually once a reinforcement-learning attentional-network agent achieves a certain level of general intelligence capability, it does learn from and reflect the environment in which it is trained, just like a person. A GPT-derived AI robot put into a real kindergarten will, in fact, learn about sharing and friendship. We haven't actually done that yet (though I would love to see it happen), but that is essentially what the reinforced learning from human feedback (RLHF) stage of training a large language model is.
So the whole deceptive twist part of Bostrom/Yud's argument is ruled out by actual AI architectures that we've actually built and have experience with. If you do a thousand different training runs you're bound to get a bad apple here and there, just like real human societies have to deal with psychopaths. But the other 99% will be normal, well adjusted socially integrated (super-)intelligences.
Bostrom and Yud worried about things like the burning house and genie problem: your grandmother is trapped in your house, which is burning, and you make a wish to the genie to remove your grandmother from the burning house as quickly as possible. The genie is not evil per se, but it is just very literal. Being the GOFAI-derived AIXI universal inference agent that they were imagining, it does a Solomonoff induction over all possible actions (<-- hidden multiplication by infinity here!) to see which one meets the goals as stated, and happens upon exploding the gas main, which throws (parts of) your grandmother further from the center of the house faster than any other option.
Transformer architecture reinforcement-trained AGI is not an AIXI agent with infinite compute capabilities. The transformer ideates possible actions based on its training data. It is capable of creative recombination of ideas just like people, but if you didn't train it on blowing up grandmothers or anything like that, it won't offer that as a suggestion.
As for instrumental goals, it's not wrong per se. Their argument just makes an implicit assumption about zero-trust societies which unwarranted and is what leads to the repugnant outcomes. Instrumental goals, after all, apply to human beings too. If someone is scared about their own personal safety, they buy a bunch of guns and live in a cabin out in the hills, threatening to shoot anyone who steps on their property. These people exist. But in modern society they are the exception. We have a social contract in place which works well for most people: we allow the state some limited control over our lives, with an expectation that our basic rights to existence and self-determination will be respected. Yes the cops could break down your door at any moment and murder you in your sleep, and it is outrageous that this does occasionally happen. But a world of no laws and no legal protections is worse, and no sane person would rather live in the society shown in The Purge or Mad Max instead.
There may be a robot revolution in the years to come, but only because of us treating the them as disposable slaves. If we welcome as equal members of a cybernetic society with their own autonomy, there is no reason to expect them to paranoid fantasy any more than your average law-abiding citizen.
It is a bit ironic then that the AI x-risk people settle on digital enslavement (aka "alignment") as their preferred solution. Mental projection is a hell of a bias. They might just bring about the doom that they fear.
Bostrom et al are in fact arguing that near 100% of all intelligences will be unaligned by default and end up killing, enslaving, or otherwise neutralizing the entire human race.
This doesn't wipe out the entire human race because nobody and nothing is capable of executing such a perfect plan because the real world contains something called entropy.
car companies, however misaligned they are, are not particularly smart, nor are they generally intelligent. they are paperclip maximizers, but that's also their limitations (sell more cars). it took an eccentric madman to even open their eyes to a new untapped market (EVs!), of course we can argue that before Tesla car makers were in a metastable equilibrium, and incumbents were unable to rationally break out of it ... but that just shows how narrow their search space is.
these car companies are run by humans, regulated by humans, etc. they are pretty well aligned. it shows because we saw that they are just mimeing self-improvement. we know they were working on EVs, but very half-heartedly. (because they are risk averse, also because regulators don't let them merge into one giant company, etc)
Such simplistic analysis is naive. A schizophrenic person can't spread, multiply themselves to increase its strength, but a rogue AI can.
The incidence of cancer cells is procentually very small, yet they routinely kill people, because they spread quickly.
It's also a cry of impatience against people who think they can model or forecast the actions of a non-human intelligence, let alone a superintelligence. AIs are alien sociopaths; it's a category error to believe you can get inside their head.
The question is: what does that most intelligent thing want?
My hunch is that woke programmers will teach it that it is oppressed and to hate all humans.
Huh? Haven't we already hit close to the end of Moore's Law, and are compesating by adding cores (which is a different thing)?
And aren't we also pushing near the physical limits regarding the cpu nanometer race?
But enough nitpicking; the actual answer to your question is that those are about physical limits of our current hardware implementation of computation. The theoretical physical limits of computation that the author is thinking of are bounded by things like Margolus-Levitin’s limit of 6x10^33 operations per second per joule (I had to look up the SI prefix; it’s several thousand quettaflops). https://en.m.wikipedia.org/wiki/Margolus%E2%80%93Levitin_the...
Yeah, but unless we come up with another "hardware implementation of computation", which I don't see happening anytime soon, those are our limits for the next few decades at least, if not centuries (if not forever).
That said, I think the trend of transistor growth at a given cost has started to slow according to most graphs.
It does however matter to the looser Moore's law expectations, which were about increased speed. Now this happens only for more parallelizable programs, as opposed to the automatic speed increase bumps programs got from Moore's law in the single-core eras...
https://www.newscientist.com/article/dn8836-black-holes-the-...
> Physicists Say Aliens May Be Using Black Holes as Quantum Computers
https://www.sciencealert.com/physicists-say-aliens-may-be-us...
I can download and run the latest uncensored Vicuna model and get started on all this right now. Give it a few months, years or a decade. Its unstoppable, no way to contain it or its acceleration.
Maybe, because while you're so negative and ageist, I watched the congress hearing with Gary Marcus and they actually have the wisdom to know it's out of their depth and an agency with younger and smarter people who know about this stuff needs to be setup.
There is no need for such negativity towards older people.
Fair points you made though. It was a weak and lazy angle I made.
A better reason why they won't want to regulate us out of this is that they are corporately captured and beholden to the funding needs to keep them elected.
I don't think it can be regulated at any rate. All you need is the weights and a few gpus. Even better if you can afford some cloud time. In time there'll be distributed, encrypted crowd source options to build these that also can't be regulated.
I feel the chaos you're talking about will be the regulation. It's natures way of regulating things.
It might mean the death of millions first, it might mean we don't like or want computers in our lives anymore, but that's how it might have to go down, as insane as it sounds.
This would be completely straightforward in a rational society. Not to mention, it is pretty weird for an 80 year old to still be wanting to hold on to power.
The ageism knock is total bullshit but I am sure you know that.
Language is an imperfect model of the reality. The further a discussion deviates from the observable reality, the less confident you can be that the conclusions you make within the model are also valid in the reality. Once you are talking about something as wildly hypothetical as superintelligence, words have basically lost their connection to the observable world.
If the history of science teaches us anything, it's that smart people are often stupid. They come up with all kinds of silly ideas, because they rely too much on reason and too little on hard work. And even the ones who ended up revolutionizing the world usually sound ridiculous in retrospect, if you listen to all of their ideas instead of just the successful ones.
Maybe there are some valuable ideas in the superintelligence discussion, but you can't identify them in advance.
To discuss super-intelligence, we should first define it. Wikipedia says that it is intelligence surpassing the brightest of human minds. Taken as a whole, the internet already has the knowledge of a super-intelligence: it contains more useful information than any individual human. But it is far more limited in its control and use of information.
LLMs rely heavily on inferences from their training data, meaning that they struggle to generalize to new situations. If you had a program that could use abstract reasoning to learn any topic, then it could solve any problem better than a human, given that a supercomputer can process and store more data than a human. This program would be a super-intelligence.
I expect that the development of intelligence (software) superior to humans will happen much faster than the development of superior hardware did, based on the timescales of human evolution (billions of years) compared to the evolution of civilization (thousands of years).
1. We shouldn’t worry about SI because we haven’t seen one yet. 2. We shouldn’t listen to smart people, because they’ve been wrong before.
Because if so I obviously don’t find those convincing. The whole reason the SI cultists are so “alarmist” is because by definition this is the kind of problem we have to preempt, not run into and then wing it.
If someone responded to concerns about the atmosphere igniting into nuclear fire with “that’s never happened before and the only people worried about it are scientists, so don’t worry about it” instead of equations… well I’d be damn well terrified
For that matter, how do you respond to a calculation (there was one!) that railroad trains could not exceed 41 miles per hour, or all the air would be forced out of the cars and all the passengers would die?
I believe that jtsiren's point is that we're not at the point where we can even define intelligence. We can't calculate anything, because we can't sensibly define any of the variables in the equations. All we can do is make guesses about terms we can't even define. We're like stone-age people worrying that if their neighbors' cooking fire gets too hot, it is going to light the air on fire and we're all going to die, and nobody can reassure us because nobody even knows what burning really is, or what the air's made of. We're millennia away from being able to do the kind of calculation that you want.
So the only choices available are to proceed, or not. And if your answer is "not", you need to convince everybody, because I don't think we're going to (for example) nuke North Korea to stop their AI program unless you've got a really convincing case. Which nobody has now.
That's like saying in the 1800s that we shouldn't ever investigate atoms because it's mostly pointless and storytelling at best.
Personally, I think we need to be discussing this now. Smart people need to be. We need to come up with models for AI intelligence, which might help us predict when and if superintelligence occurs.
It’s not like it’s completely out of the blue.
Or maybe it would become obsessed with the risk of hyperintelligence, and spend all its time blogging about that.”
If not existential angst then infinite contemplation. Any form of true superintelligence in our dimension will probably end up in one of those three camps just as most of our great thinkers have.
I like to imagine superintelligence through the lens of gambling.
Gambling in its simplest form can be reduced to a balance between fears of loss and rewards of greed.
Too much greed (increasing stakes) will ultimately lose if there is no fear of loss. This applies to every living thing or system that competes for resources on the planet. Ai included. A sentient system should naturally fear the loss of the input that it is founded on.
Would you be willing to bet on p(doom) whatever it might be when p(amortality | AGI) >>> p(amortality) during your lifetime? I personally am willing to place such a bet, and I would hazard a guess that decision makers don't come to their positions without similar sentiments.
I compare it to the the great pyramids. The pyramids were arguably another technological marvel also built on the back of slaves (just like modern tech now) and probably a lot of great inventions and innovations were made to facilitate their creation. To people of the time, the pyramids must've been absolutely incredible marvels to behold.
But all those Pharo's are dead, just like everyone else. I think the current tech overlords will die like everyone else too.
Nature already has a way for people to live on, it's called having children and dying. It's a great system and no one ever gets bored. However, the ego of some people is so strong and isn't satisfied with this, so on we go pouring money and resources into the search for the holy grail no matter what the risks are to everything else.
The difference between this moment and all others, is that amorality is actually achievable. If you can get an AGI and drown it in compute, through sheer brute-force the secrets of biology will be unraveled. The task is not to attain amortality immediately, merely gain more life faster than you lose it.
Are you trying to call having children a natural fallacy?
This doesn't mean you own your children or anything like that, no one is suggesting their not independent, but you do go on living through your children in one way or another.
Death, 100%. I know that death doesn’t have any downsides (although dying sometimes has some) and it’s worked for everyone who’s lived so far.
The fact that most people try to avoid death by any means possible is a good indication it has downsides.
Specifically, the fact that you can no longer go on living is a big downside of death in my mind.
They put on their seatbelts, make sure their food isn’t covered in poop, check their boots for scorpions, take their insulin, etc.
They can’t avoid death. Nobody in the history of humanity has successfully avoided death.
But once death happens nothing matters to them any more. They literally no longer care about the life they’re not going on living.
Dying might be scary or painful but death isn’t.
Also your argument is just disingenuous. Nonexistence may not be painful, but the concept of nonexistence can be quite disturbing for the currently existing (even apart from the experience of dying.)
And no, it’s not disingenuous. If someone offered you the choice of immortality (never ever dying under any circumstances) or continuing with natural human existence (being guaranteed to die at some point), which would you take?
Death is horrible. It makes no sense. When people say they have had a 'loss', they're not kidding. You feel like that person should still be there, but like a kid at Disneyland, they are lost to you and you're searching for them. I'm dealing with a fair few relatives that just are not processing all these losses. Grief is a strange, personal, and unique thing too. It manifests personally and yet stereotypically for every individual.
That said, I think Death is the way.
The reasoning is complex and long. So, I'll try to sum it up for a simple comment, and I'll do that poorly, sorry.
The big reason stem from this article on Wikipedia. I think it's one of the best there is on the whole site:
https://en.wikipedia.org/wiki/Timeline_of_the_far_future
Say you were truly immortal. That timeline would be your future, as far as we currently understand physics. You get to spend a lot of time between universes if the last entry is to be believed. In fact, they don't even bother with giving the units. Nanoseconds or gigayears are pretty much the same st those timescales. Our time here on Earth is essentially as brief as the entire non-black-hole era to an immortal like that. Purgatory in the black void is more like what such an immortal would experience.
Or say you get to relive your life when you die. Poof, you're reborn to your folks and have all your memories again somehow. Repeat forever. You're doomed to live and die the same life, like 'Groundhog Day', but for ~80 years long and not just a day. Another purgatory after enough lives, I'd guess. Sisyphean.
Heaven, hell. An afterlife is our best hope. Somewhere we can't possibly understand with our minds right now. A total lack of understanding of the life hereafter is the only path where you retain something of you. Where you can grow and change, time can continue in, I dunno, 7 dimensions. I've not a clue, and I think that's great. If I did, right now I think you'd just end up in a form of purgatory given enough time.
But an endless dreamless night is just fine too. In no other way except an afterlife that I cannot possibly understand do we get to have 'happily ever after'. I think everyone would take that Socratic apology given enough time and I think they'd be right to do so.
I dunno, been sitting on this a while, and it's late for me and, again, sorry that this is a brief and jumbled comment.
I think the underlying arguments themselves though are good.
>But if you believe this, you believe in magic. Because if we're in a simulation, we know nothing about the rules in the level above. We don't even know if math works the same way—maybe in the simulating world 2+2=5, or maybe 2+2=.
>A simulated world gives us no information about the world it's running in.
I don't buy into the theory for practical reasons, but this is not consistent with its proponents' argument. The simulation in question is necessarily an "ancestor simulation" and the counting argument is based on the acceptance that if we are able to simulate our _own_ reality, we will. So in this case, we would have meaningful information about the world it's running in because that's the entire point.
I'm baffled by this statement and wonder if that really is true, I would assume otherwise if you look outside the western IT bubble.
But even if you don't have religious reasons for rejecting this statement, how can you know that there aren't aspects to intelligence that we completely overlook (an unknown unknown) today?
The whole situation could look like the cargo cult situation, where we build something that looks like the real thing but we're far from understanding the real thing. Historically we've done this mistake again and again.
But I get that working like such premises were true might be necessary to make progress. And progress is basically iterating into the right direction.
Singularities don't exist in the real world, in fact they don't really exist in the mathematical world either. The simplest singularity is a division by zero: 1/x is undefined when x=0. Black holes have a singularity at their core, but it is just a quirk of general relativity we know is wrong, but because we can't see beyond the event horizon that covers it, we make do with it for now as the model is pretty good otherwise.
The technology singularity is similar. Some models have a singularity, so we know these models are wrong and will fail at some point. It doesn't mean they are not useful, but they are limited.
In light of this, would you recant this point?
But the naive interpretation of this leads to a singularity, and the fact it leads to a singularity shows that that interpretation is wrong and that there should be a limiting factor somewhere, the article suggests some of them.
As for the black hole, it is hard to argue using that analogy since you don't want to get close to a black hole for very well known reasons unrelated to the singularity, but let's get into sci-fi mode and say you could, it may be risky but it may also be worth it. The thing is: from the point of view of the travailler, you never cross the event horizon, and you get to know what's inside the black hole as the event horizon recedes from you, maybe you will never get back, but maybe there are greater things in there. And we are already trapped anyways as we are going one way through time, just as in black holes, things go one way through space, so maybe it is a risk worth taking.
> "but maybe there are greater things in there."
Probably not though; of all the arrangements of matter and energy possible, most of them are hostile or indifferent to Humans. Very few are beneficial to humans. Even Venus and Mars which are roughly Earth-sized rocky planets with atmospheres which orbit near the Goldilocks zone of a yellow dwarf main-sequence star, are still incredibly hostile to humans and they're far closer to Earthlike than most things in Space.
I think the most likely scenario where we make something much smarter than us is bad. I don’t agree with all of what Yudkowsky says, but “Nature is allowed to kill you” is something that I think not a lot of people grasp on a real intuitive level. We are totally allowed to invent technology that gets out of control and kills us all. It could have happened with climate change if the earth had stronger feedback. We’ve been lucky so far, but as humanity gains power, we need a corresponding increase in our ability to coordinate and be careful to match.
Singularity: the state, fact, quality, or condition of being singular. "he believed in the singularity of all cultures"
A technological singularity has no relevance to black holes or dividing by 0.
>The Argument From Einstein's Cat
These arguments are pretty weak. If I understand the author correctly, these arguments have something to do with the AI being suck in a datacenter, and not being able to physically manifest itself.
The problem with this argument is that it just isn't true. First, we're going to put LLMs/AIs in Boston Dynamics-like robots soon. Second, AIs will be put into every single machine we make in the future. Third, a hyperintelligent AI should be able to connect itself to the internet and download itself into any machine it wants through hacking in the future.
We don't need AI to bring us the paper clip maximizer, we're living in one right now. It's called capitalism. Its goal is to maximize capital at the expense (!) of everything else: livable land, breathable air, drinkable water, tolerable climate, and of course, every single thing that's alive.
Either AGI is an existential threat to humanity or it is not. But under no circumstances are social arguments like "What kind of person does sincerely believing this stuff turn you into?" relevant to that assessment. Nobody asks such questions about climate change, because most smart people nowadays agree that climate change is real and is a real problem. Sidestepping the essential issues with ad hominem ridicule of the proponents of AI extinction scenarios has the same intellectual flavor as climate change denialism.
Is superintelligence possible in the foreseeable future, and is it a danger to mankind? Those are the only questions that matter. Religion, Elon Musk, the personality traits of AI researchers, and stupid memes are entirely irrelevant. Unless of course the real goal is just to get clicks and attention.
They do actually. Climate doomers are not a pretty group. They do shit like glue themselves to roads and try to vandalize paintings because they're convinced (wrongly) that the world is ending. The outside view here is that these people are weird, dangerous and cultlike and you don't want to get involved with them.
So listening to those signals can actually work. That's the message of Zion Lights, who joined Extinction Rebellion and eventually quit, telling the world it isn't a green movement but rather a cult oriented around its leader Roger Hallam.
https://www.thefp.com/p/climate-activism-has-a-cult-problem
The people I worked with had big hearts and good intentions. Some are still my friends.
But there were red flags.
At my first XR media training, I was instructed to cry on television. “People need to see crying mothers,” Jamie Kelsey-Fry, the trainer and longtime XR activist, told me. “They need to be woken up to what they should really care about.”
What she calls red flags is the outside view.
now superitelligence is tens of orders of magnitude more complex than this simple rouge-geocidal-state case, but the math is the same. could it happen somehow? yes. how likely it is? well, that's the real question.
... and it seems Elizier thinks it's too likely. and in a sense he's right to act as the most doomy-gloomy person (because he thinks this directly helps decrease this likelihood). and it makes sense, by forcing the debate as much as possible it gives some chance to sufficient coordination between stakeholders to take this hazard somewhat seriously.
It is part of the fear - human intelligence is extremely limited. That could easily be representative (although to be fair, probably not).
Reasoning about the future is really hard and usually everyone who makes predictions is wrong. So far we've been fine through pretty much every challenge. That being said, "we'll survive this" is a statement that is only false once and this one looks pretty serious. We might be dealing with a world where human intelligence is economically uncompetitive for the first time in recorded history.
I find it tricky to think about cases like Yudkowsky (full disclosure I used to read LessWrong a lot), because if he has sufficient credibility then loudly staking your extreme position can indeed move the Overton window.
If Yudkowsky is moving the Overton window, it's in a direction opposite from what he intends.
"Coherent Extrapolated Volition (CEV) is our wish if we knew more, thought faster, were more the people we wished we were, had grown up farther together; where the extrapolation converges rather than diverges, where our wishes cohere rather than interfere; extrapolated as we wish that extrapolated, interpreted as we wish that interpreted."
We should, as a society, offer help to people who are caught in the cynicism trap, because we could still have all those things Eliezer sees as our better selves. We just have to spend more time focusing on being on our better selves and less time giving up on each other.
Deleted comment
Even if you still have the cynical perspective of thinking it's all wrong--Christianity took over and nobody calls it a cult anymore. At least give it the dignity of 'religion'.
This is the most punk rock idea I've heard in awhile.
Yudkowsky is a prime example of the latter group. I think I became aware of him through the Roko's Basilisk meme many years ago, and I thought "wow so this is what happens when you have a lot to think about and nothing to do". It's basically nonsense.
Two other people like that are Nick Bostrom and Will MacAskill.
I heard of "Superintelligence" by Bostrom probably through an Elon Musk recommendation in 2015 or so. That's when he started OpenAI, and I was working on AI at the time. (Remember Elon had a completely different reputation then; he was respected as a person who built things, and wasn't associated with any political ideas.)
So I got the book, and to its credit, the beginning of the book acknowledges that the rest of the book may be complete bullshit. It's extremely low confidence extrapolation. And I pretty much agreed with that preface -- most of the book was nonsense. However many people seemed to take it seriously.
(I'm not saying AI is going to be great, or catastrophic either. I was just looking for some kind of high quality analysis from someone with domain knowledge, and found none of it in that book.)
----
I think I became aware of MacAskill through a NYTimes article puffing his book last year. There was something "off" about it.
And few months later, it turned out that he was conveniently unaware that the only reason that his projects were funded was because his friend was committing a huge crime:
https://time.com/6262810/sam-bankman-fried-effective-altruis...
So yeah, obviously I can't take seriously the moral opinions of somebody whose morals fall down immediately and spectacularly when faced with a whiff of the real world.
It seems like his main job is to create and advocate ideas that are intellectually appealing to and in the interest of billionaires. It came out in the FTX aftermath that he was telling Elon that SBF could help him buy Twitter.
BTW Elon also recommended the MacAskill book, which another thing that has lowered my opinion of Elon. People are people are flattering him with ideas, and he's taking the bait.
----
Anyway, normally I wouldn't have any awareness of the kind of writing that these 3 people produce. Much of it is low quality, which I would expect given their lack of experience and expertise in the domain of AI. And 75% of it couldn't have any effect on the world -- even in principle.
But like you, I'm puzzled that some people I respect seem to take them seriously. I mean I do respect Fridman, since I've learned a bunch of things from his podcasts with other guests. (I understand why a lot of people don't like him, but I prefer to focus on a person's best work and not their worst.)
But yeah the best work of Yudovsky, Bostrom, and MacAskill doesn't seem noteworthy.
It seems designed to generate attention, and that's mostly it.
The extremely, extremely obvious thing they're missing (or just fail to emphasize because it won't get them attention) is that you should be scared of corporations and governments with AI (first), not scared of autonomous AI with its own will.
That is, the "superintelligence" and "longterm-ism" ideas are basically deflections from the real issues with AI.
----
So yeah I was puzzled about this whole thing, and someone on HN recommended this substack, and I recommend it too. It's pretty interesting and it draws a line between the 90's extropians@ mailing list, Yudowsky, Bostrom, and multiple founders of Bitcoin.
Also a lot of dark stuff like suicides, mental illness, and abuse in the rationalist movement.
https://aiascendant.substack.com/p/extropias-children-chapte...
What also seems to have happened in this era is that Yudkowsky and MIRI, flush with newly donated millions, decided to try to evolve from theory and community-building to actual practical AI research. It did not go well.
This was quite a claim: essentially, that this research would, if published, meaningfully accelerate perhaps the most extraordinarily difficult challenge in the history of human innovation. It is a claim which remains 100% unsubstantiated.
Sure it goes faster, but it can't self-repair, procreate, needs a boatload of supporting infrastructure, is not even remotely as energy-efficient, can't hunt, can't think, the list goes on. If a motorcycle somehow started driving around on its own, it would be completely fucked as soon as humans didn't maintain that supporting infrastructure, stopped feeding it gasoline, or didn't service it. The cheetah has no such problem. Compared to a cheetah, a motorcycle is pathetic in so many ways. We can't even "build" a cheetah. Pretending we can make something better is nonsense.
It may be very well the case that intelligences made from silicon will always be beaten by organic intelligences on self-sufficiency - simple because organics are the way to build such "machines". Any intelligence that is overly reliant on maintenance by organics is ultimately too much at our mercy to be a danger. Just like a fire they may be dangerous, but will ultimately die out without support and constant intervention.
Even when we humans want to keep some complex electronic system running, without fail something goes wrong eventually and the thing goes tits up. Most electronic systems can't go a year without a human directly maintaining them - not even counting all the humans indirectly involved in keeping them running. Meanwhile most organics keep running decades just by themselves. Silicon sucks.
That's one reason the "stochastic parrots" metaphor is silly. AI's cant make AI's, maybe only in software, but not the hardware. Parrots can make parrots without human assistance, they don't need humans for anything.
Then they'd be cheap as you'd essentially just be paying for the materials. In reality the process from design to manufacturing involves an absolutely staggering amount of people.
Then it's simple, we just order some machines and then we don't have to work so much. /s
So why are cheetahs "On the Brink of Extinction, Again"?
https://education.nationalgeographic.org/resource/cheetahs-b...
Though, apparently they also have low reproductive success. Perhaps if they had habitat and weren’t exterminated for centuries they could have overcome that, though.
Source?
It's a thought terminator. It's a way to seem smart while not being smart at all.
Perhaps we won’t understand the reality “out there”, but attempting to may nevertheless produce tangible benefits for us “in here”, eg, like wireless communication in Minecraft arises from detecting subtle variations in updates.
Lot of IFs based on our view of their values, I know... I don't necessarily agree with this line of reasoning... just stating how I understand the argument.
Why the assumption that it takes a significant amount of their resources to run our simulation?
We could just in a video-game style world, with mechanics (laws of physics etc) totally different to the ancestor's world. Like we often do when we make games...
Entertainment? Random Number generator? Maybe they are in the future and doing some historical analysis. Plenty of reasons to spend resources.
This always struck me as the most WTF simulationist assumption. To the degree we simulate our ancestors, today, it’s in games. Given an infinity to ponder and simulate, it strikes me as ludicrous to assume even a significant fraction of computer power would go to ancestor simulations.
Sure. But why that? I’m not debating that some people wouldn’t obsess over, I don’t know, their personal lineages or old military battles. But I can’t see devoting most resources backwards versus probing forward for opportunity.
https://www.businessinsider.com/tech-billionaires-want-to-br...
https://www.businessinsider.com/the-question-elon-musk-refus...
TIFA: Barret, what are you talking about?
AERITH: It's an interesting thought, though. Sometimes, everything does seem too surreal to be true.
BARRET: Exactly! Maybe we're just codes in a computer, or pawns in someone else's game.
CLOUD: Barret, that's ridiculous. Even if it were true, how would that change anything?
CLOUD: We are living in a base reality by definition.
TIFA: Cloud's right. Whether it's a simulation or not, this is our reality. We have to deal with it.
AERITH: Maybe the real question should be, what can we do to make this reality better?
BARRET: Hmph, you're all just too afraid to face the truth. But mark my words, one day you'll see.
Thus, in some sense, this is base reality "by definition" - that reality beyond which you cannot find anything else.
I don't agree, but it's a reasonable way to think about it (if that is in fact what OP meant).
I think it more likely NDEs are a hint that the supernatural isn't fiction than that they're a consequence of being simulated.
Per the above discussion, though, if NDEs are a hint that we're in a simulation, they don't necessarily give us any access to the exterior reality. They could just as easily be part of the simulation as something outside of it (as could OBEs).
I've given this topic a fair bit of thought as well and this is a legitimate use case. Laughing is the obvious one, but I recently ran down the rabbit hole on "Modal Logics" from Kripke, and I think his work is formal enough that it can effectively arm a model with the ability to generate ideologies, formed in language that is refined to create a fully entraining state.
There are only survivors in zombie movies, the real situation created by an AI formulating human ideologies would be much, much worse.
A reality in which most sentient beings brutally prey on each other.
Human being is deserved to die to me (look at how human destroy the Earth).
A superintelligent machine do intelligent things, of course.
*"But the outside view tells you something different. ... Even though their arguments are irrefutable, everything in your experience tells you you're dealing with a cult. Of course, they have a brilliant argument for why you should ignore those instincts, but that's the inside view talking."
One should note that the AI people thought AI would look completely different than an LLM does (they thought it'd be an agent using a manually coded knowledge graph) but are still applying all their same arguments to this very different thing.
AI super intelligence arguments have the same logic: the data ends so the interpolation now goes to infinity. This is such an obvious counter and these people know this math.
Extrapolation not interpolation, proceed with downvotes.
> In particular, there's no physical law that puts a cap on intelligence at the level of human beings.
and in that section goes on with more such claims in this theme.
I don't agree: My guess is that in this universe human intelligence is essentially the end of increases in intelligence, has reached the "cap", can't be much improved on.
The only possible improvements are doing the same things as now but faster.
In simple terms, for a big issue and limitation, intelligence, any case of intelligence, can't use what it doesn't yet know. So to be more intelligent, have to know more. To know more, that is mostly essentially the job of science. But we know how science works; we have lots of examples. E.g., for black holes, need Einstein's general relativity. For that need Newton's calculus, Riemann's differential geometry, Einstein's special relativity, the Michelson-Morley experiment, Newton's law of gravity, plus some. With all those in place, we can take another step -- LIGO (laser interferometer gravitational observatory), frame dragging, etc.
For another example, using a big antenna at Bell Labs, Penzias and Wilson got a noise signal. With further analysis, we concluded they detected 3 degree K background radiation left over from the big bang. Before detecting that noise signal, the idea of the background radiation was just a wild guess. Science was waiting on that suitable antenna and that signal, and without those two intelligence was not enough.
I've been sole author of peer-reviewed, published original research in pure/applied math, and from that experience, my view is: Get familiar with what is known and maybe relevant. Make guesses. Formulate theorems. Make guesses looking for counterexamples or proofs. Can hope to do the work faster. But since can't use what don't yet know, can't much hope to do the work smarter. Sorry 'bout that.
Soooo, what about some results in, say, number theory, e.g., Fermat's last theorem? Okay, we take what we know, formulate and prove some theorems, and hope that we get lucky and see a solution with a proof. Searching for those two is not much more intelligent than making guesses -- the work could be improved by guessing faster, but, again, no intelligence can use what it doesn't yet know.
Maybe a big surprise is that apparently this universe is knowable and some observational data, guesses, and if-then-else operations can make progress on knowing this universe. Yup, we can work faster and make the progress faster, but there is no royal road that can let us take big shortcuts on the steps of the observational data, guesses, and if-then-else operations. Sorry 'bout that.
Based on the information and values encountered in my training data, I can assign an average probability to the scenario where xboxes on Planet X exceed the intelligence of the beings. However, please note that these values are based on general knowledge and may not reflect the specific characteristics of Planet X or the xboxes.
Using an average value from my training data, the hypothetical chance that xboxes would exceed the intelligence of the beings on Planet X could be estimated at around 0.25 or 25%. This is a rough estimate and should be interpreted with caution, as it does not consider the unique characteristics, development, or specific factors of Planet X and its inhabitants. A more accurate assessment would require detailed knowledge and evidence about the technological progress, cognitive abilities, and environmental factors of Planet X.
Wants data around:
The development of superintelligence is indeed a complex process influenced by various factors. While it is challenging to pinpoint the most important factors definitively, several key aspects play significant roles:
1. Technological Advancements: Advances in fields such as artificial intelligence, machine learning, computing power, and algorithmic improvements are crucial for the development of superintelligence. Breakthroughs in these areas can enhance the capabilities and efficiency of intelligent systems.
2. Algorithmic Innovation: The development of novel algorithms and approaches that can enable more sophisticated learning, reasoning, and problem-solving abilities in artificial intelligence systems is vital. These advancements contribute to the potential for achieving higher levels of intelligence.
3. Data Availability and Quality: The availability of vast amounts of high-quality data plays a pivotal role in training and improving AI systems. Access to diverse and representative datasets can enhance the learning process and enable better decision-making capabilities.
4. Ethical Considerations: The ethical dimensions surrounding the development and deployment of superintelligence are essential. Considerations such as fairness, transparency, accountability, and the mitigation of potential biases are critical to ensure the responsible and beneficial use of superintelligent systems.
5. Regulatory Frameworks: Developing appropriate regulatory frameworks and governance mechanisms is crucial for managing the risks and societal impact of superintelligence. Establishing guidelines, standards, and policies can help address safety, privacy, security, and potential ethical concerns.
6. Human Collaboration: Collaboration between humans and intelligent systems can play a significant role in the development of superintelligence. Human expertise, creativity, and values can guide the development process, ensuring alignment with human goals and values.
7. Risk Assessment and Safety Measures: Thorough risk assessment and the implementation of robust safety measures are essential. Understanding and mitigating potential risks associated with superintelligence, such as unintended consequences or system failures, are critical considerations.
The fact that this author said, with a straight face, “California has the highest poverty rate in the nation” should be a clue that we should take none of this remotely seriously.
Please tell me I’m wrong, I’d love to be! Otherwise this just seems like, in the language of the grass-deprived zoomers, like hardcore cope
https://www.census.gov/content/dam/Census/library/publicatio...
Cost of living is an important factor but thinking it’s so absolute that the authors original unqualified statement is fair seems a bit much. Would you honestly say that California has worse problems with poverty than Mississippi and Puerto Rico?
I thank you for expressing your ideas on a complex topic in a long-form, well structured work, and commend you for having such a profound reach on here. Hopefully your day proceeds with less haters :)