This is true only if you adhere to the common definition of "unit test" where a unit test must test a single class or method on a class, often with use of lots of mocks.

Common but not necessarily best.

Let me pose a question: if refactoring is "changing code while tests stay green", then how do you refactor when any change to public methods always breaks tests? When "extract class" means a change to test class structure as well?

Testing at this close-coupled level far from mandatory. IMHO, the use of the word "unit" is deliberately vague: if it was always a "class test" or "method test" then it would be called "method test" and "class test" not "unit".

"unit" means "you choose". it means "a chunk of useful functionality", in problem domain or business terms. It means "can I test this in isolation from e.g. file system, http services and other concurrent tests", as per Michael Feathers's guidelines (1)

I'm not suggesting that you should never unit test at a close level of class and method; I am suggesting that you might benefit from a majority of unit tests being not at this close level.

1) https://www.artima.com/weblogs/viewpost.jsp?thread=126923