It's actually a myth that men have a huge advantage over women in "any physical sport."

Individuals with high levels of testosterone have an advantage in SOME physical sports. Running, sure. Sports which require size, yes. Powerlifting, yes, but note that the trans woman powerlifter in the Olympics wasn't a contender.

Many physical sports do not advantage testosterone levels. Equestrianism, auto driving (which is a bloody physical sport requiring huge levels of endurance and is dominated by men solely because of tradition), target shooting (which may advantage cis females), darts.

Cis women are better at free diving and open water swimming.

Most of the advantage of testosterone goes away when a trans woman goes through medical transition. Their times generally go down in sports in which testosterone gives an advantage.

So the concept is hard to understand because it's not as simple as you think it is.

And elite sports aren't and never will be fair anyway. People have genetic advantages, economic advantages (which I would love to see evened out more), etc.