Racket is slow which impacts the entire dev cycle, has even less use of ad-hoc polymorphism than CL (very annoying to have to use the monomorphic functions all the time), editor support isn't great (SLIME still contends well here, the speed is an issue here too), and the package story is a bit more difficult to get familiar with. Not as interactive in dev as CL, because safety.
Cool arrows in Dr.Racket though, nice macro system, and so on.