The way I do my TDD is by writing the unit tests to specifically test the functionality of the code.
For example, if I'm writing a module that does validation for a specific type (for example, in JS) I tend to test the functionality while writing the tests. I am unable to speak amongst others, but you have to test your code eventually, and it's either going to be in the classes that are using the code, the REPL, or it's going to be in your unit test/integration framework. I usually pick the test framework first because it serves 2 benefits:
1) I can prove my code works. (I must prove this to my self, or the people using the code at some point.)
2) I can reliably change the code later.
Complexity is also mentioned in a couple of comments here. TDD helps simplify complexity by writing code that is testable. It also makes you think about your API as you're writing the different possibilities the module/code can be used in.
Further, I think there are multiple issues with tests, and depending on the type of test your doing, different problems can arrise. You have your unit tests, and then you have your integration tests.
Unit tests are relatively easy to write (if your code is split into individual chunks, that do a specific task.) and they should always be written regardless of the time you have.
I think the major issues arrise in terms of time are the integration tests. When you're testing the functionality between complicated modules that require databases, i/o writes, network communication, it is sometimes hard to write the tests, and it may not even be worth it.
The majority of people who disregard TDD, usually disregard a specific sub-problem of TDD. I think TDD has its benefits depending on what type of tests you're doing, and how easily the problems can be solved with those types of tests. You can always pick to write unit tests, and disregard integration tests, etc.
I think there are a lot more issues to expand upon, like the language you're using, the platform you're using and how often your code is required to change, but TDD has large benefits, and code rot is very real and TDD can help mitigate it.