Look at your computer screen. Now move your head a foot to the side and look at it. Yes there will be some difference but unless you have a privacy filter it should be quite easy to view the screen still.

Same for the rear view screen/mirror assembly. The viewing angle at which it it's fine to watch in screen-mode is considerable. In mirror-mode it has to be very close to function as intended.

My SO is about a head shorter than me, and likes to sit in a quite different position. Yet for us the screen-mode works fine without adjustment. When using the mirror there's a significant adjustment needed. I guess for other couples it might need adjustment regardless, but I imagine for many it will be fine in screen-mode.

Though I have no idea how much it varies between cars though. Could be ours has a wider viewing angle than other cars.