Here's a common interaction, which would create that feeling without anything at all underlying it:
1. A person is sitting at a table, with me sitting at a table behind them -- both of us facing the same way, ie, me facing their back.
2. I was staring a few feet to the side of the person -- zoning out in space, and not looking at anything besides the wall pattern.
3. That person has a transient feeling someone is watching, and turns to look.
4. The motion of their head turning causes me to leap to alert, as there's suddenly motion in my peripheral vision -- I move my eyes to focus on the motion, and resolve what it is.
5. We make eye contact, validating their initial belief.
6. The other person has taken a step to condition themselves, similar to Skinner's dancing pigeons. Repeating this interaction over decades leaves them believing in a sense they don't actually possess.
Add in some bias in what people remember, eg the times they make eye contact instead of the many times no one was looking, and it's a powerful way to end up confused about the world.
I'm not sure how you'd suss this one out without an extra interaction which necessarily confounds the results (eg, an extra observer -- but since we're testing observer effects, that's messy and a half).