Augmented Reality headsets will be the "every day use" thing. After trying out Microsoft Hololens, I am very very impressed with what these kind of headsets can bring to the table.
Imagine walking through an AR doorway overlaid in your home, where you can see a beach on the other side. You walk through and magically are on said beach. You turn around and the door is gone--the illusion is complete.
More realistically, I see AR increasingly taking the form of your "main screen" with notifications, browser windows, etc. all accessible from there and just taking up visual space in your FOV. If you decide you want to focus on work or engage in another more immersive experience (movie watching, gaming, etc.), you slip into VR mode and suddenly the distractions are gone.
Again, lots needs to happen to get us there, but I see the blending of the devices as inevitable once the form factor is popularized. My guess is it will take something as sexy and revolutionary as the first iPhone to get this to go mainstream since you are right in that most people will not carry a giant headset around.
Yes. The games and immersive videos are cool, but I'm even more looking forward to replacing almost all dedicated displays with a pair of normal-looking glasses.