During performance, sheet music is like a cheat sheet during exam, a reminder of something you already know. Most of the music is in your head and hand. The sheet music is just there to prevent a memory slip. Some orchestral music can be awfully repetitive, and having the page there helps you keep track of where you are. You are not supposed to devote lots of attention to the page itself, (unless you didn't practice before the performance and decides to sightread on stage, which, you know, happens).
I will also say that learning to play without looking at your hands is a great skill to have, and also takes off some of the cognitive load.
I've played the piano since a very young age and the thing is, learning to play without looking at the keys (eg whilst reading) is actually a good skill to have. The argument goes that this way you can look at one hand without worrying about missing with the other while in a concert. I tend to agree.
Also I scribble a lot in my sheet music because part of studying a piece is discovering things written in it (everything has a purpose, every staccato, forte, piano, etc.)
Maybe because I was taught to read music for piano as a kid?
The only exception was learning banjo (as an adult) because I basically learned it all from youtube and forced myself to memorize songs. But when I could play piano I only ever remembered a few bars from a couple of songs, everything else had to be written down. Even just playing chords on a uke!