There's an unobvious upside: HFTs generate liquidity. Liquidity is good for everyone in the end.
If I put in a market order when the price is $100, and a HFT firm sees my order, beats me to market, buys up all the $100 offers and then resells the stock to me at $105, my liquidity hasn't been improved in the slightest. All that has happened is that someone has front-run my order and cost me $5/share, extracting profits from my pocket without providing any useful service to anyone.
If you want this comment in more technical terms, here it is:
http://www.zerohedge.com/news/2015-08-25/cutting-through-hft...
Flash Boys is an entertaining write-up of how the firms are inserting themselves, and how they are "seeing" orders at one exchange before the order arrives at another exchange.
The "Flash Boys" case is a funny one. The HFTs did see the orders--but so did EVERYONE else, and in basically the same instant, because all transactions are publicly disseminated. Your point is right in a very narrow sense and very wrong on every axis that actually matters.
I'm mystified as to why Flash Boys didn't valorize high-frequency trading. It's a Moneyball story where the scrappy nerds use computers and math to out-compete the dinosaurs. Oh well.
A lot of HFT argumentation only makes sense when you assume prices only go up.