Lisp is not, by any reasonable stretch of the imagination, fast. The key is that most applications aren't CPU bound anymore.

If speed is that important, use foreign function calls (or, better yet, Haskell).

And I say this as a long time CL coder who has written several highly perfomant lisp libraries. See, for example, polygon.lisp, color.lisp, and bitmap.lisp at http://github.com/smanek/ga/tree/master. In order to get decent performance you have to give up many of CL's advantages (not using half of CLOS's features, providing type hints, bit twiddling, imperative style, etc).