But things go from "everything is fine" to disaster in an extremely short period of time. A person remotely monitoring to take over in case of emergency would still have to be paying full attention and alert for the entire duration of the trip.

People are terrible at being "on active standby" like that. If they have to be really alert to the situation, it's better if they're really driving the car themselves than expecting them to recognize a bad situation and take over in time.