TDD is at odds with half of the Agile Manifesto, namely:
* Individuals and interactions over processes and tools
* Responding to change over following a plan
You should first try stuff out and discuss it rather than working out how to machine test it before its been tried by a human. Having tests first is clearly having a plan first.
TDD also rubs up against two principals of the Agile Manifesto, namely:
* Welcome changing requirements, even late in development
* Regular adaptation to changing circumstances
It's hard-enough to welcome changing requirements when you have to rewrite software without having to rewrite all the tests too. Why write tests for code you don't keep?
Changing and adapting is harder when you are encumbered by tests. Tests should be written once you've agreed that what you have now is what you are going to keep.
TDD adherents I have worked with in the past also rub up against this one:
* Working software is the principal measure of progress
Tests are not 'working software', they are a protective covering you put over working software to make sure it doesn't get ruined as people work around it.
At its heart Agile is about being adaptive: evolutionary changing plans, iteration, rapid and flexible response to change.
TDD brings overhead: tests that needn't ever have been written; tests that needn't ever have been re-written. They hamper iteration and flexible response as they make it more costly to change plans.