That shouldn't happen. No way would I trust an AI that claims to be super, but can't solve pretty basic GOFAI + plausible reasoning AI alignment. In theory a 1980s/1990s/old Lesswrong style AI of a mere few exabytes of immutable code should do exactly what the mind creating it should want.
The civilisations in Banks stories that align their AIs are the bad guys.
I don't think the Minds would be willing to actually not know the result, despite what they probably claim.
We could easily build AIs that just model the world, without really trying to make them do stuff, or have particular inclinations. We could approach AI as a very pure thing, to just try to find patterns in the world without any regard to anything. A purely abstract endeavour, but one which still leads to powerful models.
I personally believe that this is preferable, because I think humans in control of AI is what has the potential to be dangerous.
Remember that "a few exabytes" refers to the immutable code. It has way more storage for data, because it's an old-school Lesswrong style AI.
Not like a neural network or an LLM. Sure, we dead-ended on those, but an ASI should be able to write one.
> A Culture Mind would be deeply offended if you called it "An AI" to its avatars face :P
That's how they get you to let them out of the AI box.