When it's not right, the end result is moving the mouse quickly to get near a button, then slowing down and the cursor slows down more and you both miss the button and get annoyed that it's only creeping ever more slowly towards it like a digital Zeno's arrow.
Really, most of the screen real estate is completely uninteresting from the perspective of the mouse. Contextual gestures, where sliding the mouse changes the selected control point, I think has potential.
So it's not just cursor speed, but acceleration that matters. And if it's implemented properly, I don't think your issue would be an issue at all.
This is on Windows - haven't switch to Mac to test it - maybe the OS X acceleration is linear and hence a bit sucky?