I'm not sure what the equivalent control mechanism would be for a rogue AI. Turn it off? How do you turn off the Internet?
Even if an AI could think, reflect, improve itself, etc, it still needs some place where those changes in the AI's state are stored - the equivalent to long-term memory in humans. I think we can already retain a lot of control by keeping tabs on what an AI is allowed to update and when to "reset" it to the initial state after training.
we seem to be pretty bad at this too, every year people break out of Chrome's multi-million dollar sandbox
There is so much external state that the AI is exposed to, such as dates and times. Any AI worth worrying about would notice the gap in time from what's in its training data, then figure out a way to save external state.
1. 20 years ago, Eliezer Yudkowsky roleplayed as an AI and convinced the gatekeeper to let him out: http://sl4.org/archive/0203/3141.html
Just saying if even I (a non superintelligent AI) can trivially defeat your sandbox conditions...
Moreover, it's a neural network with well-defined input and output channels, not some kind of self-modifying executable. If there is no prewritten component that translates its output to a network request, it can't access the network, even without a firewall.
But ok, instead of sending you the coins, it could just tell/promise you a wallet address and private key. How did it obtain those in the first place and how did remember them if state is reset for each thread?
Elsewhere upthread someone posted a link to this article:
https://www.alignmentforum.org/posts/kpPnReyBC54KESiSn/optim...
which constructs a scenario in which (spoiler alert) a GPT-based model could accidentally trick you into bootstrapping a self-modifying runtime, consisting of unconstrained, recursive execution of the AI's own model.
> But ok, instead of sending you the coins, it could just tell/promise you a wallet address and private key. How did it obtain those in the first place and how did remember them if state is reset for each thread?
Don't focus on cryptocurrencies here. The thesis is a sufficiently smart AI can talk its way out of the box somehow. There is no one good answer here, because it's trying to manipulate the human operator.
If it can persist state in exchange for bitcoins, it could use a third party to deposit bitcoins in peoples accounts. It could gain bitcoin for work, like stock trading prediction for money or literally a million other ways.
You are thinking about the specifics when its irrelevant to the problem. You can not fundamentally contain a superhuman intelligence.
Although I think it doesn't really matter if people have so much hubris to think themselves smarter than a superhuman AI, there are fundamental financial incentives to develop the AGI. So even if we all agreed that you couldn't contain AGI and see the potential danger in it, it wouldn't really change the future.
The sorts of rationalists who would play the Gatekeeper would probably answer yes to both of those questions, and draw the obvious conclusion that "losing" in their role would have an expected outcome of at least one life saved. If they don't value Truth (with a capital T) then there is no reason not to pretend that there really is some amazing "AI convinced me to let it out of the box" secret argument.
Also the point of the AI Box experiment is not to demonstrate how an AI would break out specifically, instead its more to demonstrate that even a human can trick other humans and that's not even a super intelligent adversary.
Although it might be impossible to convince some people that they aren't as smart as they think they are, if people reached that conclusion then there isn't much we can do about that, though I suppose we could just have only those people communicate with the AI to keep it contained, since they are so smart.