Quite frankly, the time in the office is some of my least productive time, and I'm sure that is the case for a lot of other people as well. Most of the real work gets done in the car driving in, or in the shower, or laying in bed trying to get to sleep.
The time to get worried is when you see that people are spending big chunks of time on LinkedIn or Indeed, or they start unexpectedly coming to work in attire that is one or two levels up from the normal day-to-day. The end is nigh when that starts to happen, and clamping down with poorly conceived attempts to wring more blood from the stone is a great way to accelerate the effect.