I think "shame" is a needlessly inciteful description. The main issue is that, in a company with multiple employees, your behavior is not happening in isolation. When one person is working 80 hours a week, it is very common for management to see everyone who chooses to structure their time differently (e.g. because they have kids or elderly relatives to care for) as a slacker.
If it turns out that management is not thinking this way in your company, then great! But I think most people commenting skeptically here are familiar with situations where one over-achiever in the team sets an impossibly high standard for everyone else and thus unwittingly causes them harm.
This is also why it is usually not considered a problem when a person pursuing their individual passion project puts in crazy hours. If you work for yourself, you're not setting standards for anyone else.