Your experience tells me this is possible, at least, when management has no visibility into your real productivity. Either through lack of interest or time to supervise results - the construction site manager. Or through lack of skills - the software job where you did pointless refactoring.

At least the construction site manager had a point - morale is contagious, and you being seen sitting around by other, busy workers would not have been good for their morale.