But the main issue for me is having to switch between layouts. Actually, it's an issue I used to have living in France, where I occasionally had to switch between my QWERTY layout and some AZERTY layout (for instance, to show something to a colleague and friend). Even though I know both layout, there is always some adaptation time.
Slightly unrelated, I noticed that what breaks my typing flow is to add/remove pairs parenthesis/braces (which happens all the time when coding). I still haven't found a good solution to this. I think that even Vim doesn't have a good default solution for this. I wonder how efficient typers solve this issue.