That isn't the key point. Taking hormone therapy as an adult doesn't erase the huge athletic advantage conferred by going through male puberty.
i.e., unless something fundamentally changes about how leagues are divided, there's going to be perceived unfairness in sports.
As long as things are unchanged, I think the real conversation boils down to who we prioritize: cis women or trans women.