I think the big mismatch here is that romantic desirability is largely determined not by someone's subconscious assessment of their future happiness with a potential partner but rather their perception that the potential partner would help them produce successful offspring. A conventionally attractive partner might not make you happier, but they are more likely to be healthy and therefore to pass on healthy genes to your children.

But over the last 50ish years in the west, we've begun to expect more from our romantic relationships, including many different, potentially inconsistent objectives beyond just the production and raising of children. We want friendship, intimacy, meaning, happiness, etc. And romantic relationships where the production of children is not a goal, and in some cases is not even possible, are increasingly common too (eg many--but not all--LGBTQ+ relationships, child free hetero marriages, and so on).

Not that this is a bad thing, but on some level we are still monkeys driven by the need to procreate and we are following those same subconscious instincts even when they are inconsistent with our conscious desires to form long-term happy relationships.