These speed issues are annoying but the thing that kills my experience is the lost clicks - and when one is missed, because it's quite common that Windows is just being its usual useless self, you then don't know for sure if it really missed the click or not. Invariably the time you think "it did miss it, try again", you'll click again only to find that now your slow Windows machine is stupidly struggling to do the damn task twice!
The other issue doesn't sit completely with Windows: in a corporate environment, there are numerous remote activities that get hooked up without much thought or care and it only takes an occasional slow response with one or two of them for Windows to become unusable.