There is a fundamental philosophical error in that assumption. Goals are not set by sheer rationality but emotionally. Logic is a guide to the means to the end and to give wisdom about how well pursuing a given goal may work but it is ultimately the emotional which sets the drive.

An AI life coach cannot guide you for if you would prefer having a large family, being part of a DINK couple, or remain single for the rest of your life. It can tell you the odds of success of living off of a band are low and to prepare backup plans,and that being a chartered accountant is a more reliable source of income but it cannot tell you what the right decision is to attempt.

Yeah, but what are you optimizing for? Optimal only means something if it’s in context.

The human decision maker still wins if you’re optimising your life for novelty.