Stop caring what other people think of you. The ego is the problem.
We have a name for people who don't care what other people think: narcissists.
What I think you're getting at is that you can't please everyone, and there are many cases where risking someone's opinion of you is the right thing to do.