I've seen this claim multiple times and it's never made sense to me. I do TDD on side projects where there's not even a hint of "agile."
If you think using TDD improves the quality of your code, then it's also important for maintaining high code quality, since that enables some of the other agile practices, such as frequent releases.
I don't worry too much about whether a particular practice is agile or not. If you find TDD useful, then great! It's often included in conversations about agile development, but that doesn't mean you have to be doing agile development in order to use it.
Agile warns against building stuff you don't need. TDD means that test provide the justification for feature work. If your feature work isn't making a failing test past, you don't need that feature work. Obviously this assumes that your tests are justified (ideally there is a stack of tests leading all the way to the end user experiment), but that's much easier to get right than justifying untested feature work.