Yes. Male puberty confers lifelong physical advantages on the body in terms of muscle mass, heart size, lung capacity and bone density. That's why men dominate women physically in every sport, which is why women have their own special sporting categories so that they stand a chance. Hormone therapy reduces these advantage but it doesn't eliminate them.
> Is it fair to trans athletes?
Probably not, but then is it fair to wheelchair-bound athletes that they're physically incapable of competing with Olympians and so need their own separate event? Maybe the cruel reality of life is that we can't always accommodate everybody?