I'm no financial expert, but I believe HFT benefits many people by making the difference between buy-price and sell-price smaller and increasing the liquidity of the market.
The banks seem to be the only ones benefiting from this as far as I can tell. My life doesn't feel any better for HFT.
Look around you. Pick any product you like. A thousand financial transactions went into making it, all along the supply chain. Currencies were converted and hedged, iron ore and copper and crude oil futures were purchased and sold. When technology squeezes out middlemen and makes them more efficient, you gain.
On the other hand when bankers manipulate prices then everyone looses.
I could be wrong, but from what I understand of HFT, it doesn't squeeze out middlemen, it squeezes out competition. It sets the barrier to entry so high that only the super rich can participate. I agree completely with your point, by the way, I'm just not sure HFT is a good example of it.