I don't feel guilty for over-sleeping. I've worked out a life for myself where people don't care that I sleep until 11:00. I simply don't schedule meetings or appointments for the morning, or if there's absolutely one that can't be avoided, I go to bed early and set an alarm clock to wake up for it.
It is not being a sissy to know your limitations and preferences and to work around them. If you make a commitment to another person, you should keep it. But nobody is forcing you to make that commitment in the first place. Is it really impossible to schedule that meeting in the afternoon, or to not have it at all?
It's not the OPs place to say what sort of sleep habits other people should or should not have. If he finds them annoying, fine: don't work with them.