I've heard the term used in a purely negative sense, while personally thinking it's often necessary and sometimes brilliant to do some yak shaving. Some of the best yak shavers I know are essential people, even in a "move fast" startup environment, because they effect significant change and improvement that can lift the whole team out of a local maximum of productivity.

Maybe for people who use it negatively, "refactoring" or "unplanned improving" isn't "yak shaving" until you've gone beyond what's reasonable. E.g. if you're doing some recursive refactoring, and improving the code base, great, but yak shaving is when you've truly gone off course.

Either way, I think this is where healthy team dynamics are so useful, because it's hard for the yak shaver to know if they've gone off course or are doing something unplanned but useful. If I think I'm deep in a yak shaving exercise, but I think the results will be beneficial, I announce that I'm yak shaving and explain why. If the rest of the team thinks it isn't truly a useful exercise, they can guide me out of it.