The biggest adoption problem is quality VR games (Beat saber is really the only true stand out hit), followed by comfort and form factor of the hardware, followed by marketing a perception of VR. I don't think VR will ever be a smartphone or PC sized market but assuming current growth is maintained it will be a small game console sized market (10-20 million units) in a few years. This could be seen as a failure or a future that never arrived because VR won't be truly mainstream, but for all major players except Facebook (Sony, Valve) I think it will be viewed as a success.
AR hardware should be separated out as well because I think the challenges there are also applications but also technical difficulty. The mythical always on high resolution glasses or contact form factor display that could be a smartphone replacement is a lot longer away than advertised I think. If and onc technical and design challenges are reasonably solved AR hardware could take off.