The other issue is just the amount of incompatible VR ecosystems out there, Cardboard, Daydream, PSVR1, PSVR2, Rift, Go, Quest are all separate system with little or no compatibility. That means you constantly end up with good content that only a fraction of the VR market can access, which is a huge problem when VR is already short on content.
Finally, VR never managed to integrate with the rest of the gaming world. Ports of regular games into VR were actively discouraged in the early days and only a few of them ever happened. That meant the whole promise of VR allowing you to "step into your games" fell flat very early on. None of your favorite games could be revisited in VR, instead you where stuck with a lot low budget indie tech demos. Lately that has been changing with UEVR, which allows to mod VR support into Unreal Engine games, but that still is just an unofficial hack, not a feature embraced by any of the VR companies.
There is of course also the big "motion sickness" bogeyman in the corner and while that can certainly be an issue, most people just get used to it. I think the efforts to mitigate the issue (don't port games to VR, make everything slow and boring) have done far more damage to VR than they helped, since it simply means we are stuck with a whole lot of games in VR that simply can't compete with what's available in the world of non-VR gaming. That's kind of a big problem when VR was always supposed to be the "next-level" of gaming and yet it feels like a step backwards.