I've discovered recently that its the video hardware and how that interacts with the precise (sic) choice of graphics driver that governs Unity speed.
cletus: you can test my hypothesis by logging in with the Unity 2d session and seeing if the UI becomes more responsive. Scaling (super-s) will be 'ragged' without 3d effects but should be faster.
drivebyacct2: what is your graphics card and what driver? I need a new desktop box soon myself...
I'm giving Unity a try and I have to admit its a little fussy and busy. My fallback is Debian with a tiling window manager (dwm) so I've been going in the 'menuless' direction for some time.