And the main reason for developing AIs is to do things in the real world. So, unlike a prisoner in jail, the AI might be advising the government or helping scientific research, which would provide many opportunities for subtle manipulation, even while it is imprisoned
If you can make an AI that's 100x smarter than you are, surely you can make one that's only 2x or 3x smarter. Why not start with the latter? If the experiment turns out well, then you can use the 2x/3x smarter AI to guard a 10x smarter AI, and so on.
> And the main reason for developing AIs is to do things in the real world. So, unlike a prisoner in jail, the AI might be advising the government or helping scientific research, which would provide many opportunities for subtle manipulation, even while it is imprisoned
Most of the time, it is much harder to come up with solutions to a problem than to verify that the solutions are correct. If you consider NP problems, for instance, we can use the AI to find good solutions heuristically and we can prove the correctness of their solutions with a trivial algorithm. That would still be very useful, but there wouldn't be any room for foul play.
How do you quantify human intelligence like that anyway? We are very similar to proposed AI/machine learning techniques in that we form predictive models from observation, but how can that really be measured?
With what resources? The AI is in a box, remember: we are running it on a machine with only so much memory and processing power. There's only so much it can do within these limitations, it's not like we're giving it a spending account.