I've been suspicious as well about that at first, but after having tried Google Cardboard myself on a OnePlus 3, I was surprised by how accurate accelerometer powered head tracking worked.
I'm sure there are specific cases that aren't covered by this level of accuracy, but overall, it works perfectly fine.