Isn't the point of TDD that you can change implementation at will, safe in the knowledge that the tests will help guarantee that you're not inadvertently changing behavior?
Most of the time you have classes that interact with one another. If you test each class individually, and you decide to refactor what classes do what, you'll need to rewrite a lot of tests, even though the higher level behavior should not change.
Usually however, what I see (in Java and Spring Boot land) is test methods which use mocks that are set up to expect certain method calls, e.g. when(accountService.getAccount(id)).thenReturn(account). Typically there are anywhere from six to 20 lines of this stuff per test (and, it's rarely de-duplicated into a separate method). So as soon as the contract to getAccount() changes, a bunch of tests need to change.
Second, it's common to believe that the "unit" in a "unit test" is either a class or (less often) a method. Consequently, people write test classes for every single class of a group of classes that are acting in collaboration, when what they really ought to do is write tests against the single class supplying the public interface, varying input to that class as necessary to achieve good confidence.
If you do not have TDD, I guess the reasoning is that either you do not have tests for this (probably bad) and if you do, less of them (and so more easily changed).
Not sure it's an argument against TDD, but I guess if management/the programmer do not know about sunk cost fallacy, it might make them hold on to bad abstractions/layers.