You can set hotkeys to go directly to a specific virtual desktop. I have ctrl-1 though ctrl-9 setup for the reason you called out.