That is my experience too.
But it might be something that is subjectively felt, and people then attributed it to be something that depletes. Even if the depletion is invalidated, the subjective feeling might however still be valid. --- If I should suggest another possibility on the spot, I would suggest mental fatigue. You get tired of denying yourself things the same way you get tired of denying your kid pestering you for a treat.
It's not directly depletion, but I could subjectively describe it as a resource getting depleted.