The full expression this term is taken from is: "a few bad apples rot the whole barrel".
Women don't know in advance who the bad apples are amongst men, so in contexts where women are particularly vulnerable to predation from such men, the whole barrel must be considered as being rotten. This is why we have sex-segregated spaces.
Treating someone who seems like a woman, acts like a woman, is at risk from the gender-based violence inflicted upon women… as a man… is making people safer, how exactly?
How would you even determine which women to exclude? If someone's saying "hey, I'm trans, I'll leave if my existence makes you uncomfortable", they're clearly not one of the Evil™ Ones… so are you proposing to check everyone who enters these spaces? Oh, but wait, the state of medical treatment is actually pretty good, these days. Better make that a thorough check. (No. Just, no.)
And how is it a good idea to put trans men in with women victims of (usually male-perpetrated) domestic violence? Isn't the whole point of the "women's spaces" thing to avoid that? (Or do you propose to just ban all trans people from support services entirely?)
To be fair to the OP, I don't think there exists a history of abuse of females from trans men, which strengthens their allegation that the underlying sex might be an important factor to consider in these scenarios. The sex hormones alone maybe just aren't a factor.
It's likely not safe to place trans women into male prisons either though, so I think there ought to another axis than just male/female along which we can segregate, such as violent/non-violent criminal history/tendencies. Come to think of it, that seems almost obvious in retrospect. Put violent offenders together, and non-violent ones together, and then non-violent offenders are far less likely to become harder criminals upon release.
Male-on-male violence is a problem for the male prison estate to solve within its walls. It doesn't make any sense to put women at increased risk of male violence to protect men from male violence. Women don't exist as some sort of violence shield to protect men from each other.
Men's prisons already have safeguards in place for vulnerable male prisoners, such as paedophiles, ex-police officers, ex-gang members and so on - putting them on separate wings of the prison, moving them to a different category of male prison (which is in practice similar to your suggestion of violent/non-violent separation). If men who call themselves women are at elevated risk from other men, the same safeguards can be applied to them. There is no reason whatsoever to house them in a women's prison.
"Underlying sex" isn't a particularly coherent concept to begin with. By what potential mechanism could "underlying sex" influence behaviour, other than hormonally?
Lacking an answer to this question doesn't mean we should rule out the possibility while investigating, but we shouldn't take too much heed of the possibility until we have a reason to believe it. «"Excess male violence in prisons" problem is a consequence of the patriarchy» is a concise explanation, introducing no additional entities or mechanisms, consistent with the observations mentioned in this thread, and… well, as far as I'm aware, it's the mainstream feminist position on this matter.
> Come to think of it, that seems almost obvious in retrospect.
I've had thoughts like that, on occasion. Unfortunately, segregationist policies are a much more politically feasible approach than anything like what you're proposing, and I don't think that will change any time soon. How to segregate is the question: whether to segregate is a pretty radical question to be asking.
I don't think your proposal is quite perfect, but it's a start.
By that argument, trans people shouldn't exist because they have the hormonal profile of their birth sex, therefore they shouldn't exhibit observable behavioural differences.