I wouldn't underestimate the impact of "social" gaming on wii's success, it was seen as accessible, casual, gaming at an affordable price. One of the unfortunate aspects of current gen consoles is that the games don't support single-console multiplayer for the most part. I have to leave an go home if I want to play destiny with my friends.
With VR you can't really share the experience so it sort of loses the word of mouth type virality that came along with Wii where you might go to someone's house and play together. Instead it's a solo experience. Even if multiple people have headsets (even more money) you still aren't sharing in the experience in any way.
The smartphone headsets are somewhat better, but still just kind of a neat trick. I don't deny that VR is plenty cool but the barriers for widespread adoption are just too high in every direction I look.