The real threats for Apple, imho, are successful web companies expanding towards hardware. Apple does great HW and poor online services; someone doing great online services and great HW could hit them badly. Google got close already; Microsoft will try again once they complete their retooling; Amazon are unfocused but could fish a winner with some original thinking; Facebook/Dropbox/others might also challenge at some point. Luckily for Apple, hardware is very difficult to get right and get profitable.
VR isn't going to be a novelty if people switch to experiencing life primarily in a virtual manner due to the vast expanse it will offer. That includes travel, art, gaming, social, shopping, media.
With today's technology? No. With the technology we'll have in 30 years? Yep. And particularly the technology that will gradually evolve over that time. It'll be far more impressive and useful than the smartphones we have today, in every regard.
If you had told someone in 2003 that billions of people would use their cell phones for so much today, and that people would practically be attached at the hip 24/7 to them, I don't think you'd have been believed - the notion that everyone would have their face buried in large glass screens for hours each day, socializing, shopping, etc.
I'd rather follow a set of virtual arrows to my destination than have to keep checking my phone. This goes for a lot of tasks I use my phone for. Though the privacy concerns are more worrisome than mobile phones!
I don't think Apple has ever been as clear on a new product segment.
If Apple gets so far behind that they need to offer it to their customers, they will probably make it easier to integrate Amazon, Microsoft, or Facebook's offerings.