There are two things: having privilege, and wielding it over people. If you are white and male and have a good job, you have privilege and there's nothing wrong with that. No one really has a right to check you for that.
If you then go on to tell all kinds of people that you got where you are by working hard and everyone else just has to work hard and they'll get there too, you're minimizing their challenges. They have to work harder to maintain a realistic, positive attitude with respect to the hurdles in front of them, which makes their life harder. Meanwhile, your life is not affected at all by the conversation, you go back to work and are productive. That's wielding your privilege, and that's a dick move, and people have every right to check you for that.
As for the difference between privilege and luck or blessings... Privilege is a kind of continuous luck.... one's maleness pays off throughout their entire life. Finding a dollar on the street is only instantaneous luck. And privilege only applies to things that are substantial advantages in the sort of common denominator pursuits of the median individual. Your maleness helps you earn more money which is something pretty much everyone needs to do. If you have small wrists and can easily clean out the inside of glass jars, that's not really privilege.
As for why you hate the word, perhaps you hate it because you think being sexist and racist means you're evil, therefore having privilege makes you evil. But everyone is sexist sometimes and everyone is racist sometimes. It's harmful, but it doesn't mean you're evil. It's just the way people work. That said, recognizing your racism and sexism and seeking out ways to be less so is an opportunity to be good.
As for people making you feel guilty... No one has a right to make you feel guilty. If they're trying to make you feel bad, they're just being an asshole. That said, they could be simultaneously being an asshole AND checking you on something where you're being an asshole at the same time. If so, I would just forgive them (to yourself) for trying to make you feel bad, and thank them (to their face) for helping you understand better the ways in which you're harming people. Learn something and don't let the petty stuff get in your way.