I agree that optimizing for "respond to all things promptly" could lead to what you're saying. That doesn't need to be what you're optimizing for, solely.
If being good at email brings more high value email, well, that's one of those good problems maybe. If it brings more low value email, treat the low value email differently. Or rather, treat all email as what it is.
I'm not saying there's no place at all for purposefully delayed responses.
-- Donald Knuth
People who do that go to the bottom of my "would hire again" list. I like working with people who communicate in a timely manner. Nothing wrong with replying, "I' busy today, I'll get back to you after the weekend" if you want to get off the hamster wheel of replying in a timely manner without being a jerk.