I think that's what makes driving fun.. you have greater control of your cars actions and are in touch with it. I could never see myself driving something where I never have to touch the shifter once its in a forward gear.
As for the speed issue, you're right, it is slowing it down. (granted new paddle systems make that 60 milliseconds like the F430 Scuderia) But the Tesla only has 1 gear, which doesn't like going much past 100mph.