This discussion boggles my mind, though I'm willing to assume there's something to it. The reason I love my unit tests so much is that they free me to refactor endlessly. In fact, given that I'm working in an old code base that has no testing, the only thing letting me refactor to the extent I need to accomplish my current task is the unit tests I'm adding.

And I'm not spouting propaganda here, as I know this is a talking point, this is my personal experience. Take away this benefit and my desire to write the tests would be much less.

Before unit tests, I would do design one, build a bit, discover I need design two, build a bit more, discover I need design three, build a bit more, discover I need design four but now design three is really embedded too deeply into the system to know whether I can safely change things, and then the hacks start going in because otherwise the subtle bugs start popping up faster than I can catch or fix them. Unit tests don't eliminate the bugs popping up unexpectedly after a design change, but in my experience it brings them down far enough that the cost/benefit of the design change is positive again, which in the long term pays off when we actually end up with the rather well-tuned, powerful-yet-simple design thirteen. (Someday I hope to learn enough to get there sooner, like, design six or something.)

Am I just lucky, or are they doing something wrong?

On the front of what I do differently than some, I actually reject the "unit testing must test as small a unit as possible" dogma and the vast bulk of my unit tests are actually what some people would call integration tests, though, usually, rather low level integration tests; they don't take 20 minutes to run, either. I also avoid mock objects like the plague, aggressively preferring to either build the interface to be testable in the first place, or failing that, adding some hooks in the tested code itself, rather than trying to push it onto the code from the outside. Perhaps that's the difference... basically I tend to run my testing one level above the bottom layer on the real-as-possible code, rather than right on the bottom layer of the code with various fake bits attached. Despite what intuition (and dogma) might be telling you, this usually means I can change that bottom layer quite significantly with little to no test code changes.

(I don't follow this religiously, there is some bottom-layer code that gets tested on-the-spot, like raw validators or the output of a parser, but typically the code I test on the bottom layer never subsequently changes much because it is so simple that there's simply no reason for it to change.)