I think it depends on the amount of information you shove into a users eyes.
The body balances by using the contours of objects seen through the eyes mixed with sensory input of the balancing nerve and touch.
So to much overlay might make you sick as well I think.