The main limitation to the AI governor model, which has been tried in many games to varying success, is that the solver necessary to make them play like an optimal player tends to start encompassing more and more parts of the game mechanics as you add in tricks like "use a worker to rush production" or "plan improvements to coincide with tech advances". The player, likely to be of a controlling or anxious mindset, will reject the idea that these details can be left up to an unaware AI, and given the option to do so they will micro-manage the fun out of the game.

When the game has relatively few options that can be played to a high depth, like Chess, of course, it's easy to write such a competent AI with a brute-force approach. But that's usually not how 4X games are structured: there are a ton of decisions involving overlapping timers and resource bonuses and force committments, and so the top-end strategy revolves around leveraging everything in a just-so way to accelerate growth and minimize losses.

The move towards lower micro and symmetric balancing in newer games has the upside of downplaying much of this; your advantage, where you get one, is much less likely to come from careful micro employed to replicate a 50% advantage in a single scenario hundreds of times, and AI benefits as well. But it also loses some of the charm of weirdly unbalanced 90's strategy.