Users are sensitive to tiny differences in speed, much tinier than most software developers consider significant. For example, there have been many studies showing dramatic drop-offs in web usage with what seem like micro-differences in response times.
A good example is how Chrome is gaining rapidly in market share primarily by being more responsive. There was a fascinating post recently on a political blog I read where the author was talking about switching to Chrome even though he couldn't quite figure out why. I read it as an example of the psychological spill-over effect of performance. (Edit: here it is: http://www.talkingpointsmemo.com/archives/2009/12/the_case_f.... Well worth reading for anyone interested in the psychology of software.)
It's a fascinating conundrum that while hardware has gotten orders of magnitude faster, software has remained sluggish. I know it's fashionable to explain this by saying "but software is doing orders of magnitude more than it used to", but I think this is only a partial explanation; it's also the case that we're leaving huge amounts on the table because of the way we write code. I used to believe in the dream of an ideal high-level language abstracted away from the hardware altogether. That now seems to me a harmful delusion. What we need is a better way to combine high-level constructs with low-level ones. This in turn (veering wildly off on a tangent here...) explains the persistent appeal of C++ despite its manifest perversity. No other mainstream language is even in that game.