I've seen this happen to myself numerous times as well, but I don't think this is good argument against TDD.
If a tests fails because the implementation details change, that tells us that we wrote a bad test, not that TDD is bad. As I get better at writing tests, I am better able to catch these kinds of errors ahead of time. There are a ton of little techniques you can use and questions you can ask yourself about your code that can make your tests better.
Furthermore, test code is just as important as production code, and we should be just as ruthless in refactoring and peer-reviewing each others tests (within reason).
TDD is NOT suitable for every problem, this is true. But I think too many people give up on it because they see "the problems of TDD" which are actually just "the problems of writing bad tests".
Note: I'm not trying to target you sago, but merely making a general observation. I'm talking more about people who don't write tests even close to the time of writing the code, or don't write any tests at all because the their tests kept breaking for no good reason and were slowing them down.