Chess are very limited compared to the whole universe, AGI could for example invent self-replicating fusion-driven paperclip factories, there's nothing in the universe rules that makes this impossible we simply don't know how to do it (yet).
But based on we actually know about the sort of AI systems humans actually know how to build and obtain useful results from, we'd end up with a computer which absorbed time series of paperclip sales figures, supply chain costs and marketing campaign data which recommends small paperclips for the Kazakh market, and there's no reason to believe a model trained to optimise paperclip production would know anything at all about how to generate fusion power. This article suggests we might even be able to analyse why the algorithm prioritised that obscure market after applying a lot more thought to the relevant data ourselves. (We can also reason that the AI model is more likely to be wrong about the small paperclips for the Kazakh market, because the permutations for the Kazakh market aren't as perfectly encapsulated in the underlying model and simulated by training on data as the permutations of chess)
> suppose we invent some superhuman AGI for $100B in 2100
not
> AI systems humans actually know how to build and obtain useful results from
You're talking about specialized expert system that is possible today.
BTW: AGI don't need to know anything about fusion power, just like Alpha Zero doesn't know anything about chess when it starts. Its supposed to learn the rules as it goes, and in case of AGI its chessboard is the universe.
Chess are a pretty well balanced and stable environment, so there's no way to cheat, but when you give less well-defined environment to AI it routinely cheats. See for example AlphaStar working around the APM limit by "saving" APM for crucial fights and completely demolishing human players in these fights microing at 1000s actions per minute. Or evolutionary systems learning to walk by abusing numerical instability bugs in the physics engine.
We know there are cheatcodes to reality (see industrial revolution), we know we haven't yet discovered all of them, and the assumption was AGI will be smarter than us. What you're doing is wishful thinking AGI won't find them cause it will be too busy with accounting.
The "cheat" argument suppports my argument, not yours. Rather than do something incredibly difficult and indirect like master advanced physics outside the scope of the simulations it can run to invent new power sources to legitimately produce very low cost paperclips, a system to maximise paperclip sales indicators is probably going to exploit gaps in the simulations it can run or data its already supplied with, like data entry errors in a region or logical holes in the supply chain pricing model, or the marketing simulation not including indications of languages spoken. So it might actually be worse than the management consultants at optimising paperclip revenues. On a related note, if you did want your AGI to invent fusion power but your fusion power emulation engine isn't up to scratch, it's likely to design reactors that are very performant inside the simulation but don't actually work, much like those evolutionary systems exploiting physics engine bugs when faced with the comparatively simple task of learning how to walk.