If there were we would see trans-athletes winning lots of medals. They don't. In fact there just aren't that many athletes. This is a giant nothing-burger.
There is another argument that says trans-athletes can be disadvantaged by having a skeletal structure which their hormone system can not properly power and this is actually hinders their athletic performance. Either way as I said, there aren't any trans athletes winning medals so I don't think we really have anything to be concerned about.
As for your concern about female sports I have my doubts about whether or not you are sincere but I'll pretend you are.
Perhaps it is easier for you to think of Female sports as "a protected category for people who do not have male levels of testosterone". This is what it effectively is anyway because of inter-sex humans. We monitor people's testosterone (whether that is right or wrong is another issue) and we decide if they are allowed compete in female sport. The exact same process can be applied to trans-athletes.