At that point, if you are doing those things, there is a certain amount of validity to the criticism that if it didn't work, you really weren't doing it right. Either something external jammed you up so you weren't able to adapt and truly follow agile, your team was personally unable to execute on the adaptation for some reason (structure, personality issues, experience levels, being simply too large to be able to be this flexible because large teams simply need more structure), or the task was simply too hard in the first place for some reason (such as "it doesn't matter how agile and smart the team is, you're not getting 4 people to produce a standards-compliant browser that is also an office suite in six months").
We're not doing a very good job of picking the right people to manage things; quite the opposite, in many cases whatever success is despite the leadercrap we have to deal with.