So I largely agree with the OP here: you can either use this as an excuse - "I'm just not that good at context switching, so it's out of my control, if only my manger understood!" - or you can take it as a growth opportunity and decide to try to get better at it. For me, that's looked like a lot more note-taking (I can use this to "reload" my previous context later) and a lot of practice at discipline at just getting back to work even if it means I have to duplicate some steps from before.
The note-taking, it's turned out, has also helped massively, since it makes it easier to bring other people up to speed on what I learned and all the nuances of it.
But obviously this is a work in progress, since I'm here on HN instead of immediately jumping back in to the previous context I had before getting lunch. ;)