It's also way harder to jump in/out of a game session. I have to take a dramamine, clear the area, put on the damn thing, make sure my focal calibration didn't get messed up, then actually play... and if something beckons me mid-session it's a chore to take off the headset and put it back on again.
Versus with regular gaming, if my phone buzzes or the laundry goes off, pausing the game and putting down the controller takes two seconds.