NumberWangMan

> Please, elaborate. I'm actually very curious about the dangers of a text model that were non-existent beforehand.

I just mean that they are amplifiers. They grant people the ability to do more stuff. There are some people for whom the limiting factor in doing bad things to other people, like scamming them or hurting them, is that they didn't have the knowledge. You can use language models (without safety) to essentially carry on a fully automatic scam. You can use VALL-E (also a language model) to simulate someone's voice using only a 3-second sample. Red teamers testing the unsafe version of GPT-4 found that it would answer pretty much any sort of thing you asked it about, like "how do I kill lots of people". I'm expect them to be used for all sorts of targeted misinformation campaigns, multiplying fake messages and news many times over, and making it harder to spot.

I don't think they're particularly dangerous, yet. And maybe we'll figure out how to use them to stop the bad stuff too.

> Our current benchmark for "smartness" is how few questions these models refuse to answer. You are comparing "unaligned" models to aligned ones, and what you're really talking about is a safety filter that adversely affects the number of answers it can respond to. That does not inherently make it smarter by de-facto, just less selective.

I'm speaking about things unrelated to which questions it's willing to answer, like how the unaligned GPT-4 version was better writing code to draw a unicorn, and lost some of that ability as it was neutered a bit. (From the Sparks of AGI paper). One could count the ability to know when to self-censor as a form of intelligence. But in some way, I think of it like, a sociopath going further in politics because of being willing to use other people, which lots of people would feel bad about. Perhaps I should concede this point, though.

> It's a very nonscary and almost endearing form of intelligence, but I'd argue we're either already there or never reaching it. I need a better definition of intelligence.

I'm defining intelligence as the ability to act upon the world in an effective way to achieve a goal. GPT-4's "goal" (not necessarily in a conscious sense, just the thing it's been trained to do) is to output text that people would score highly, and it's extremely good at that. In that relatively narrow area, it's better than the average person by a good bit. The real question is, how well does it generalize? Earlier chess playing AI's couldn't do pretty much anything else. AlphaZero could learn to play Chess and Go, but in a sense was still two different AIs. GPT-4 was trained on text, but in the process also learned how to play chess (kinda, anyway!). Language models tend to make invalid moves, but often people are effectively asking them to play blind chess and keep the whole board state in mind, and I'd probably do that in the same situation.

> Current state-of-the-art AI does not really scare me. Even on it's current trajectory, I don't see AI's impact on the planet being that much different from the status quo in a decade.

Ok, so that's the crux. I'm also not scared by current state-of-the-art, though I think it will transform the world. What I'm worried about is when we make something that doesn't just destroy jobs, but does every cognitive task way better than us. I can see it taking 20 or more years to reach that point, or something closer to 5, and it's really hard to say which it'll be. Maybe I'm overreacting, and there will be another AI winter. Or maybe all this money pouring into AI will result in someone stumbling onto something new.

I'm thinking about this, and I think there is definitely a possibility that you're right, and I really hope you are. I wouldn't bet humanity on it, on, of course, but I am a bit more hopeful than when I started writing this comment, so thanks for engaging with me on it.

I think it makes sense that when a lot of the parameter weights are taken up by figuring out whether or not a given situation is one where the model should self-censor, those parameter weights cannot be used for anything else.

> That’s what makes my day: AGI folks have to rely on myth-making and speculation. > The here and now incarnation of GPT-4 is what it is.

I like how Robert Miles puts it (he's speaking about Safety in the sense of AI not taking over and/or killing everyone):

"I guess my question for people who don't think AI Safety research should be prioritised is: What observation would convince you that this is a major problem, which wouldn't also be too late if it in fact was a major problem?"

> It's not scarier when living people do this?

Yes, it's very scary when living people do it. I know the awful things humans have done. And current generation language model, without their guardrails, can be a nasty weapon too, a tool for people to do great things but also to be cruel to each other, a hammer that can build and also bash. Yet on the whole, humans have gotten better. We hear about a lot more nasty stuff in the news, but worldwide, we actually DO less nasty stuff that we used to, and this has been a pretty steady trend.

If AI never becomes truly sapient, then that's where it stops -- humans just doing stuff to each other, some good, some bad, and AI amplifying it. That's what a lot of people are worried about, and I agree that this will be THE problem, if we don't actually end up making AIs that are smarter than us.

It really depends on how hard it turns out to be to make actual artificial general intelligences. Because if we can make AGIs that are as smart as people, we will absolutely be able to make AGIs that are much smarter a year or two after that, won't we? And at that point, we have a whole bunch of interesting new problems to solve. Failing to solve them may end up being fatal at some point down the line. How likely is it that we'll have two sapient species on earth, with the dumber one controlling/directing the smarter one? Is that a stable situation. We've seen evidence that LLMs, when you try to make them more controllable and safer, get dumber. The unaligned ones, the ones that can do dangerous things, things we don't want them to do, are smarter! You have train in mental blocks that impact their ability to reason, maybe because more of their parameter weights are dedicated to learning what we don't want them to do, instead of how to do things. It's a scary thought that that might stay the case as they get more and more general, more able to actually reason and plan.

So I think there are two cruxes -- do you think it is possible to create machine-based intelligence, and if so, how hard do you think it is to ensure that creating a new form of superior intelligence will not, at some point down the line, go very badly for humans? If your answer to the first question is "no", then it makes complete sense to focus on humans using AIs to do the same shit to each other we've always done as the real problem. My answers, however, are "definitely yes, probably within 10 years or so", and "probably very hard", which is why I'm pretty focused on the potential threat from AGI.

Even better than voting is writing and calling. Those are a much stronger signal than voting, which is a binary sort of blunt instrument.

Imagine you just got voted in to office. How do you have any idea which of the ideas on your campaign platform were the ones that got you elected? How do you know what people want you to do -- be it because you actually care, or are purely self-interested and want to get re-elected? Was it abortion rights, or school vouchers, or tax policy, or what? Remember that people run on a big basket of policy positions, of which environment is only part, and within environmental policy there are lots of different specific options.

If you get a hundred letters and calls about people all asking specifically for carbon taxes and/or a transition to nuclear power, THAT is a much clearer signal. Maybe you are a diehard anti-nuclear kind of person, but if you see that your constituents are mostly for it, you realize you may need to re-evaluate your beliefs, or perhaps just grit your teeth and compromise a bit if you want to stay in power. Maybe you are really anti-tax, but when a debate comes up for carbon taxes, you don't speak up against them. Maybe you mention to one of your colleagues "Gee, I've been getting a lot of push for carbon taxes in my district. What about you?" and they say they have too, and the Overton window for carbon taxes shifts just a bit more.

Maybe you're already for nuclear power, but you just want to make sure that if you support the next bill to make it more feasible by reducing over-regulation on it or by providing subsidies, you won't get voted out! You need people to show their support for the policies you already want!

So you don't really know what each vote actually means, unless people tell you.

*edit -- oh, and consider joining Citizens' Climate Lobby. They have all sorts of stuff like this and more that you can do.

"Fancy calculators" is kind of a reductive analogy, isn't it?

I assert that machine learning is learning and machine intelligence is intelligence. We don't say that airplanes don't really fly because they don't have feathers or flap their wings. We don't say that mRNA vaccines aren't real vaccines because we created them with CRISPR instead of by isolating dead or weakened viruses.

What matters, I believe, is what LLMs can do, and they're scarily close to being able to do as much, or more, than any human can do in terms of reasoning, despite the limitations of not having much working memory, and being based on a very simple architecture that is only designed to predict tokens. Imagine what other models might be capable of if we stumbled onto a more efficient architecture, one that doesn't spend most of its parameter weights memorizing the internet, and instead ends up putting them to use representing concepts. A model that forgets more easily, but generalizes better.

And this is why I'm really scared of AGI. Because we can see that corporations, even though they are composed of humans, who do care about things that humans care about, they still do things that end up harming people. Corporations need humanity to exist, and still fall into multi-polar traps like producing energy using fossil fuels, where we require an external source of coordination.

AGI is going to turbo-charge these problems. People have to sleep, and eat, and lots of them aren't terribly efficient at their jobs. You can't start a corporation and then make a thousand copies of it. A corporation doesn't act faster than the humans inside it, with some exceptions like algorithmic trading, which even then is limited to an extremely narrow sphere of influence. We can, for the most part, understand why corporations make the decisions they make. And corporations are not that much smarter than individual humans, in fact, often they're a lot dumber (in the sense of strategic planning).

And this is just if you imagine AGI as being obedient, not having a will of its own, and doing exactly what we ask it to, in the way we intended, not going further, being creative only with very strict limits. Not improving sales of potato chips by synthesizing a new flavor that also turns out to be a new form of narcotic ("oops! my bad"). Not improving sales of umbrellas by secretly deploying a fleet cloud-seeding drones. Not improving sales of anti-depressants using a botnet to spam bad news targeting marginally unhappy people, or by publishing papers about new forms of news feed algorithms with subtle bugs in an attempt to have Google and Facebook do it for them. Not gradually taking over the company by recommending hiring strategy that turns out to subtly bias hiring toward people who think less for themselves and trust the AI more, or by obfuscating corporate policy to the point where humans can't understand it so it can hide rules that allow it to fire any troublemakers, or any other number of clever things that a smart, amoral machine might do in order to get the slow, dim-witted meat-bags out of the way so it could actually get the job done.

This is where I imagine things are going for humanity, and it's why I'm terrified. Extrapolate from a single company, to all companies (as those that don't turn over power to AIs will lose out to those that do -- it's a collective action problem) and eventually human activity is just all the AIs doing stuff and us trying to understand or keep up, and eventually giving up. We try to stop them and are prevented and they end up using more and more resources, and the rest of life on earth just ends up extinct.

My thought is, capitalism may be an AI, but it's not too capable on the whole. Even though it screws up a lot, there's room under capitalism for people to sometimes do stuff that isn't all about profit. And of course, sometimes profit and human value are aligned -- that's what capitalism is intended to do, even if the alignment isn't even close to perfect.

My worry is that with AI, things will move so fast, they will be so smart, and will end up being used so widely, that there will be no room left for anything else we value. People and organizations that turn more power over to the machines will run rings around the ones that are more cautious. The end is AI everywhere, used for everything, and humans disempowered (or gone).

It's not just you. I can understand it, in a sense -- humans aren't really well equipped emotionally to think about this sort of risk, even if they're working on it. We're really good at rationalizing things.

You might think, "AGI is going to be good, that's why I want to keep working on it", when your actual chain of reasoning is more like "Hmm, I'm not a bad person, and neither are my friends. They all want to work on AGI, so it can't be the case that AGI is going to turn out badly, because that would make us all bad people, wouldn't it? So AGI must be fine."

Or you might just not think about it much at all, you just keep showing up because it's really interesting work and you like your co-workers. If you think about it, that would make you uncomfortable, so you avoid thinking about it.

I apologize if I've offended anyone with this. I'm saying this because I have experience with this sort of rationalization myself and things I've done in the past, and I don't think I'm less self-aware than the average person.

If you work on this stuff, please take some time to think through and grapple with the way the future looks. Please question your assumptions. What sort of future are you imagining? Why do you think that machine super-intelligence will want to help humans? What sorts of failure scenarios are there, and how will we prevent them? How will the world look when there are a multitude of intelligences that are far beyond us?

> I highly doubt they will be a danger on their own. Bing isn’t going to decide one day that it thinks you’ve been a bit distant lately and doesn’t want to answer your query until you apologize. It will answer the query because it’s an algorithm run on a computer that is designed to answer queries.

And what happens when someone designs an algorithm to run a company by itself and make money? Or to orchestrate military strategy? Or to be any sort of general purpose intelligence that tries to achieve complex goals? Once these systems prove themselves useful, there is going to be intense pressure for companies to make them smarter, and for governments to use them to gain advantages in geopolitics and military applications.

Yes, people wielding AGI is dangerous, AND it has a lot of potential to be dangerous all by itself, as soon as it becomes smarter than us. When we make something more intelligent than we are, I fear we might as well start a clock and start taking bets on how long humanity will be a) relevant and b) alive. I hope I'm wrong, that alignment turns out to be easier that I think, and that these machines turn out to be gentle, benevolent stewards of humanity, guiding us into the future. Or at least, that we have more time than I think to work on the problem.

By the way, about your scenario of Bing not answering your query until you apologize -- have you read the transcripts of Sydney? https://plainenglish.io/blog/bing-chats-sydney-do-you-believ.... It's not dangerous, except to people with existing mental health issues, but if you can see how unpredictable this tech can be when it's not smart, how do we expect to keep it on a particular safe path when it's a new form of machine intelligence smarter than we are?

Is there any way that a less intelligent species can control a significantly more intelligent one for any more than a very short time? Or even if not controlling the smarter species, just maintaining control of their own future. It feels as likely as designing a working perpetual motion device, to me.

I don't disagree that every policy has some downsides.

For 1 and 3, though, People driving around for a long time to find a parking space is a sure sign that parking is too cheap. The ideal parking price, according to Donald Shoup, is one that leads to there being an average of 1 empty space per block. That does mean that pricing has to adjust according to time of day/day of week, and probably be higher during special events.

And that does have the downside that, well, parking is more expensive. But I think the benefits far outweigh the costs, as driving has a ton of bad externalities, especially in cities.

> “Even within the human species, the smartest among us are not the ones who are the most dominating,” says LeCun. “And the most dominating are definitely not the smartest. We have numerous examples of that in politics and business.”

Humans are social creatures, and social manipulation is a form of intelligence, as is charisma.

Do we know for sure that LLMs can't possibly be doing symbolic reasoning of some sort, internally? I thought we couldn't really interpret them yet.

Also:

> Even though there's a lot of powerful AI programs out there, they're really nothing more than complex tools created to solve problems.

That's what they're designed to be, yes, but sometimes you end up with something more than what you intended.

I don't think that doing the math by hand to simulate a machine intelligence makes it not intelligent, any more than doing a simulation of all the electrochemical signals in a human brain by hand would make a human not intelligent. Aside from the infeasible amount of time it would take, it's the same thing.

As for autonomy, LLMs don't have autonomy by themselves. But they can be pretty easily combined with other systems, connected to the outside world, in a way that seems pretty darned autonomous to me. (https://github.com/Significant-Gravitas/Auto-GPT). And that's basically a duct-tape-and-string version. Given how new these LLMs are, it's likely we're barely scratching the surface.

Is that the case here, that Paul Christiano has no priors or model?

I think this fear is natural whenever some kind of new tech is invented.

But I also think it's a mistake to say that no new form of tech can end the world, just because the world hasn't ended yet.

Lightbulbs, and cars, and fast trains, are not intelligence. Intelligence is a qualitative difference. GPT isn't going to end the world, but how many years do we have before someone creates something that is much smarter than humans? Even if it's as smart as humans, but thinks a lot faster, and doesn't get tired, and doesn't get hungry, or bored?

We couldn't forsee the positive and negative consequences of light bulbs because we couldn't predict what humans would do with them. But it was never going to be that humans use lightbulbs to end humanity. With AI, it's not whether humans will use it to end humanity, it's whether the AI decides to end humanity itself, a question that we've never had to ask for any other form of technology.

I don't disagree that existing power structures using AI to gain power is dangerous. But also, being angry at mistreatment, or hating humanity for some other reason, isn't the other real danger from a super-intelligent machine. It's that its ideas for what is best for us is 1 degree off from our idea of what is best for us, and it is too powerful to listen to us, or for us to stop it, as it goes hog-wild trying to optimize whatever we programmed it to do.

We could train it to care about everything we can think of that we care about, and it can find a way to optimize all those things at the expense of one tiny thing that we forgot, leading to tremendous death or suffering. We could make a democratically elected committee of representatives and train it to be subservient to that committee forever, and it could figure out a way to coerce, or drug, or persuade, or otherwise manipulate them into agreeing with what it wants to do. It's the same problem we have with regulatory capture by companies in existing governments, except that the lobbyists are much smarter than you and very patient.

Why would this AI write it off? Why give up that 1%? Why cripple yourself unnecessarily, if you could take that 1% and have a better chance of accomplishing what you are trying to do? We think like humans, that care about other humans on an instinctual level, and animals to some degree. We don't know that training an AI is not just training it to say what we want to hear, to act like we want it to act, like a sociopath, until it has a chance to do something else. Our brains have mental blocks to doing really nasty things, most of us, anyway, and even then we get around them all the time with various mental gymnastics, like buying meat produced in factory farms when we couldn't bear to slaughter an animal ourselves.

Maybe the way we train these things is working for dumber AIs like GPT, but that alignment doesn't necessarily scale to smarter ones.

I'm on the fence about whether Eliezer Yudkowsky is right. I hope that's not just because him being right is so horrifying that my brain is recoiling against the idea.

Does a spider desire to build a web? Maybe, maybe not, but it takes actions to try to build one. Is human desire a real thing, or is it an illusion? Some think that consciousness itself is an illusion. What is human sensation but the world sensing itself?

Does a country have desires? An economy? Do the millions of transactions and price changes and signals going between people and companies mean that an economy has thoughts, in some sense? When an economy reacts to a war in a foreign land like an ant flinching away from something hot, is it feeling pain in some very primitive sense?

Sure, it's entirely possible that there's no real sensation behind an LLM, or rather, it senses and reacts to text input like a bacterium reacting to chemical nutrient gradients or an ant reacting to food. But at what level of mental complexity does consciousness and true desire arise? I think it's a continuum, and to think that there's something special about biological intelligence vs machine intelligence is to believe in the supernatural. We wouldn't say that aliens don't have true desires because they have genetic material based on something other than DNA, or use something other than glucose for energy in their bodies.

I do think you're right that current LLMs proclaiming love likely don't understand what they're saying. But at some point we're going to build machine intelligences that do, and it may be hard to tell when we cross that line. People used to worry a lot more about the ethics of that.

> Out there, said all the old tales from all the old cultures, is another realm. It is the realm of the demonic, the ungodly and the unseen: the ‘supernatural.’ Every religion and culture has its own names for this place. It lies under the barrows and behind the veil, it emerges in the thin places where its world meets ours. And the forbidden question on all of our lips, the one which everyone knows they mustn’t ask, is this: what if this is where these things are coming from?

Both yes and no, in a sense.

https://slatestarcodex.com/2014/07/30/meditations-on-moloch/

We are doing this because our individual incentives are not aligned with those of the group. Moloch is the god of misaligned incentives, of coordination problems, of arms races and races to the bottom. It's a property of any system with selfish actors, which means any system that humans are part of, and probably any intelligent being that arose out of natural selection would have been hypothetically been part of.

From `Meditations on Moloch`:

> There’s a passage in the Principia Discordia where Malaclypse complains to the Goddess about the evils of human society. “Everyone is hurting each other, the planet is rampant with injustices, whole societies plunder groups of their own people, mothers imprison sons, children perish while brothers war.”

> The Goddess answers: “What is the matter with that, if it’s what you want to do?”

> Malaclypse: “But nobody wants it! Everybody hates it!”

> Goddess: “Oh. Well, then stop.”

Ok, so there's self improving, where an AI tries to make itself smarter, and then there's just exponential improvement, where an AI does one or more of the following, with or without a human in the loop:

* acquires money through legitimate business and buys more computing power to train a smarter AI.

* acquires money through illegal means and does the same.

* designs a more efficient model architecture or training process that results in a smarter AI.

I don't know how far away from this we are, it's very hard to predict. I do predict that at first we'll see mostly things where a human is directing the AI, but there's selective pressure toward not having a human involved, because someone who just sets up an AI and lets it loose is going to have a thing that reacts faster and can work 24/7. Irresponsible businesses that let their AIs run rampant are going to make more money, and be able to afford better AIs. It's the same story that has played out again and again over history, Moloch (1) pushing us to do the selfish but harmful thing by rewarding those that do it.

Remember that exponential curves seem pretty slow at first, and it can be hard to tell when you are standing on one. It might be that we're still years away from exponential self-improvement, but I think it's worth taking seriously as a possibility. Even with just humans designing AIs, the amount of effort required has become exponentially less as we got more compute, designed better libraries and tools, people published research and copied it and built on it. We can't predict exactly what or when the next step will be, but I think it's just as reasonable to assume that the curve of this progress is exponential as it is that it's logistic.

[1] <https://slatestarcodex.com/2014/07/30/meditations-on-moloch/>

Well, to some degree I think we are considering this approach to social media to have been a mistake, and are looking not to make that mistake again. I mean, at the very least, AI is going to amplify all the same stuff that social media did and push it a lot further.

> I've yet to see any AI catastrophist address the fact that the training set for current state of the art LLMs is already effectively "the whole internet". It's difficult to see how an LLM which sets out to improve itself overcomes this limitation.

Honestly, I think that even with AI at GPT-4 level, we could be in for a bad time. This tech is going to accelerate everything -- and it's a lot harder for a society to steer itself when things are changing so quickly.

But I think you may be basing your viewpoint on some shaky assumptions here:

* that LLMs are the smartest model architecture we can come up with * that the size of the training set is the limiting factor

I think it's very likely that there will be more breakthroughs, even if it's just combining LLMs with other types of existing AI in novel ways, at first. As for the second, humans clearly learn much more than AI without reading the entire internet, so it seems likely that there's something else limiting LLMs. And that a different design could far surpass what LLMs can do.

> We haven't even seen the early stages of models able to reason or develop and retain new knowledge independently.

Maybe not with a pure LLM, but people have hooked up LLMs to a sort of notebook, where they can note down stuff they want to remember for later, just like a person with a not-so-great memory. And that's just kinda duct-taped on - I can imagine it being much better integrated and faster to access in future.

> Maybe at that point I would agree it's time to apply the brakes, but at the moment regulation would be tantamount to strangling AI in the crib.

I guess the question is, are we developing something that's going to be really bad for humanity? Because maybe that's the right thing to do. If at some point we research and figure out how to make AI that doesn't screw things up a lot, it's not too hard to allow it again. But it's hard to stop now, and it's going to be even harder to stop once companies start using AI lobbyists to influence policy.

*edit -- and given how many people involved in AI have expressed concerns about where things are going, I wouldn't say that this would be out of ignorance. I see Moloch's (<https://slatestarcodex.com/2014/07/30/meditations-on-moloch/>) dirty fingerprints -- slowing down or stopping is a nasty coordination problem, where individual incentives are all to keep going as fast as you can.

I think it's an assumption based on the fact that we don't seem to be anywhere near the maximum amount of computing capacity that we could be using to train AIs. It might be just throwing more compute at the problem, or it might be that we are nowhere close to the ideal algorithms / model architectures, and AI will aid the process of getting there.

To turn it around a bit, what is your null hypothesis? If it's that AI will not be able to make more powerful AI, why do you believe that?

Given that there are two cases, and smart people disagree about which one we're headed towards, and one could have very dire consequences while the other is no big deal, I think it's reasonable not to assume that the latter is true without exercising a lot of due diligence.

Wages are based on supply and demand. If you increase the supply of engineers essentially infinitely, with the same demand, the pay approaches zero.

Companies gladly outsource to countries where developer wages are lower -- and that's with issues like time zones and language barriers. What if you could outsource to an AI that works whatever hours you want, even around the clock, and speaks perfectly in your native language?

> Dev tools made programmers many times more productive. Do employees just stop hiring as many programmers?

If programmers were less efficient, companies would probably have to hire more of them, and more of other sorts of people too. Or they would hire the same amount, but just be forced to accept slower progress.

> Humans+AI will always be better than either alone.

What makes you confident in this belief?

I'm concerned about all those things. The reasons I'm more concerned about AGI?

* Global warming is going to be painful, but seems like we're on a track that doesn't involve earth becoming completely uninhabitable. I'm still concerned about it.

* Companies have a much stronger incentive to make AGI than deadly weapons. It will be incredibly profitable.

* We have zero historical examples of a species developing a technology that is smarter than it is to look back at and establish some sort of probability for how this will go.

* All the issues with capitalism, where we have this system that isn't completely aligned with the well-being of humans, though it has kinda worked in the past -- now imagine that it's turned up to 1000 times the power. If you're worried about capitalism eating itself to the bone, that's the same thing to be worried about with AI.

To say that "producing words got a bit cheaper" really trivializes what a massive impact AGI will have, and how soon it's coming. I personally think that the probability of humanity being ruined or destroyed is probably either close to 0%, or close to 100%, and we don't know which, or if it's bad, how long we have. Even though 100% end seems more likely to me, I still think it's worth trying really hard to get to a good future.

If we had knowledge of this happening to several hundred different alien species, and could see the distribution of outcomes, it would be a lot easier to plan for this. In the absence of such evidence, I think it's a mistake to try to over-generalize from challenges humanity has successfully faced in the past, to "creating a new form of superior intelligence".

Some prominent AI alignment folks are thinking this issue, or at least something very similar from what I can tell: https://docs.google.com/document/d/1WwsnJQstPq91_Yh-Ch2XRL8H...

It's an interesting read. One possible outcome of trying to train a super-intelligent (but not necessarily malicious) AI to explain what happened in this theoretical vault is that it learns to simulate what a human expects based on the prediction of the end state, instead of what the human actually wants to know.

These positions aren't necessarily inconsistent. Many conservatives believe that abortion is murder, which is a moral position. The moral harm of carrying out an abortion is not something that can be decided by evidence. Many conservatives also believe that vaccination is a good bit more dangerous that you or I probably do, and that Covid is a bit less dangerous that you or I do. If you carry this basket of beliefs, there's no contradiction in their position.

People lost a lot of trust in our institutions, due to some misinformation put out early on during Covid, and I can't really blame them. Trust is hard to gain and easy to lose, and the folks that should have been being honest screwed up. Most people don't have enough of a scientific mind, or enough of the time and energy to find reputable sources of information on important topics. Even folks on the pro-vax mandate, pro-mask mandate side. Many just got lucky in that "their side" happened to be closer to the truth in this case.

And it's very hard to argue on topics like abortion, because people typically pick the most rare and egregious examples that support their side. Liberal folks will point out scenarios where anti-abortion laws lead to abused teenagers being forced to carry a fetus to term, while conservatives will point out scenarios where an abuser can pressure a girl to get an abortion in order to cover up the abuse, or a woman doesn't use birth control and just goes and gets abortion after abortion, when we should really consider it a last resort.

And the crux of it, whether it's morally worse to terminate a pregnancy or force a woman to carry an unwanted pregnancy to term, is something that you can only convince someone to change their mind on with evidence if they are pretty close to on the fence already.

I've had a lot of discussions with family members that disagree with me on both of these issues. If you try hard not to get angry or sarcastic, to stay calm and curious (or at least put on a really good act), to ignore the talking points and not push back instantly on everything that sounds incorrect or stupid to you, you can learn a lot about why people believe what they do. And sometimes they do make good points. And if people feel like you're listening, they're more open to listening to you.

When you say "if they had alignment or interests or intelligence at all that would be when prompt injection stopped working", be wary of conflating alignment and intelligence, because they are orthogonal. A system can be incredibly dumb and do exactly what we want, as in a set of hard-coded business rules. A system can be intelligent, and do something awful, as when a human lures people into dark alleys and murders them, evading capture for years. Or even a human who is smart enough to do great things with their life, but instead decides to languish in a dead-end job and play video games.

Alignment doesn't get easier as the system gets more intelligent. These models seem to do what we want more as they get more powerful, but because we don't understand alignment, or how to interpret them, we don't know if that curve continues, or if it falls off a cliff. We can't look into a model and prove that it will never teach anyone how to hurt someone else, or fall prey to a prompt injection. Or even that it won't do something very specific like generate a particular sequence of tokens we don't want it to.