As someone who firmly believes in "that male privilege thing," people who tell you that you should feel bad about it are wrong. There's nothing wrong with privilege.

It's a lot like privilege in the operating systems sense. sudo isn't a bad program because it has root privileges. The kernel doesn't need to apologize for running in a privileged CPU mode. But sudo and the kernel both have abilities that regular userspace programs don't - and the ability to cause damage that regular userspace programs don't. They (or more specifically, their authors) need to be aware that they're privileged, be careful about doing things with the privilege by mistake, and realize that other programs can't do the same things they can. But that doesn't mean that they can do everything, or that it's their fault if there's something they're unable to do, and it certainly doesn't mean that it's meaningful or productive for them / their authors to feel bad about the privilege.