This means most public places are just not going to work. Even in your own home, you might be unwilling to disengage so completely if you have a mischievous sibling, an animal who might get underfoot or delicate/dangerous items near you that could be hit by a wayward swing of the arm. In general, you are completely reliant on others around you to warn you of danger - from wandering too close to walls or furniture, to watching out for others who might want to take advantage of your vulnerability.
Offices might work, but not open plan. Classrooms are a non-starter. Airplane seats would be good (if not great), but not public transport like a subway or a city bus, even if you disregard the nausea issue. Even sitting in a high-end cafe or a public library, you'd need to have someone actively looking out for you. Again, would you be comfortable laying down for a nap in the middle of a park? If not, then using VR there isn't going to work.
This is an unavoidable issue that can only be slightly mitigated with automated detection systems. There are inherent limits to a technical solution like that: Someone staring at you intently from a short distance away to gauge how they could best run up and grab your bag isn't something a proximity warning system would be able to counter. Already just concentrating on portable video games in public (phone, Switch, etc.) is such a distraction that you have to be wary of your surroundings - the idea of adding in complete sensory deprivation to it as well is farcical.
VR simply isn't the next smartphone in terms of technology. It will forever be a niche used at home, alone. That isn't to say this is a small market - the entire video game industry has a similar use case - but Zuckerberg has his hopes set on a revolution akin to the iPhone and it just isn't going to happen.
AR is a much more realistic opportunity. If Zuck was as smart as he thinks he is, he'd pivot and focus there. But I don't think it will happen until Apple launches their solution (whatever it may be), and by then it may be way too late.