I really hate people walking behind me when I'm trying to work. The feeling that someone might be behind me prevents me from fully concentrating on what I'm working on.
A previous boss liked to stand behind me and watch (and occasionally make inane suggestions) when I was trying to fix a particularly high-profile issue. It negatively impacted both the speed and quality of my work.