Volatility is the main concern [1]. Making trading decisions based on microseconds is speculation. How can anyone or anything reach an understanding of something as complex as markets in this short time?

Other concerns include unnecessary waste and unfairness [2]. The point is that this is a zero-sum game that makes us invest in infrastructure that will soon get obsolete once someone else builds a faster infrastructure.

Fast trade isn't "efficient". Long-term stability is.

[1] https://en.wikipedia.org/wiki/2010_flash_crash [2] https://en.wikipedia.org/wiki/Flash_Boys