Tesla says that the owner is supposed to be able to see the car during all parts of the summon procedure precisely to prevent situations like this. Tesla knows that this system is not perfect.
Fwiw, the best use of summon is simply getting out of a tight parking spot (e.g., some residential garages).
Expecting it to be a valet is ambitious in its current state.