Big screen TVs aren't convenient either - but a huge number of households have them - VR doesn't have to be exactly like smartphones in order to be very popular
It's unlikely people will even feel comfortable walking through the room, since they have no idea when the VR user is going to freak out and blindly start flailing body parts around reacting to something only they can see. This means the entire living room is off limits, which is also very inconvenient when compared to someone just sitting on the couch watching Netflix.
There are a lot of social hurdles that VR will need to overcome, in addition to each of the technical ones.