But words shape the images we form in our minds. The issue with man-limited phrases is that we imagine men, specifically. That's also how words work and we know this through studies.
So we ought to fix this. But I'm sure OP already know why people seek to get rid of these unwanted gendering in idioms. That's not new anymore.
It's mildly annoying because we are used to the old term but that's very temporary, for the time of the transition. But I'm sure OP instantly understood the meaning of Person in the Middle the first time you saw it, so I have troubles understanding why someone would oppose such an evolution. Surely it's not about clarity.
We never fixed anything by refusing change and staying in the past.