What a wild affliction! (no sarcasm)
We've rescued and live with four cats also, and a feral ancestry dog, and I've worked with hundreds of feral or stray cats with others with no obvious problems.
I believe what you describe and it makes me wonder, you may have just had horrendously bad luck.