My personality is a bit different than yours. For right or wrong, I used to think like that, then I realized I was allowing people to walk all over me. I no longer allow such shenanigans and I feel 500% better for it.
What has counteracted that tendency, which I think may be partly innate, partly learned (a paranoia spectrum?), is two things: 1) the only reason anybody ever wants to tear me down is to build themselves up, and having been guilty of that myself, I can have compassion for them, and 2) I have found that I can put faith in the universe that, so long as I do the right thing, I go to bed with a sense of dignity and am not so afraid. That's not to say I don't have boundaries. I can have boundaries without taking it personally when others cross them, though.