A TV is like a piece of furniture, or a decoration like a painting. As compelling as it is, you can choose to ignore it, make eye contact with and talk to someone else in the room wether the TV is on or not. It's casual.
I imagine too what an outrageously high pixel count a VR set would have to have to render a virtual TV with the same resolution as a real one.
And even aside from the above, I really can't see myself putting something on my face and dropping out of my environ for anything more than about 5 or 10 minutes. It's just weird.