I know there are flaws to my reasoning, but I can't shake the thought: How/Why those poor people ended on the track is more important to fix than to focus on the responsibility of the guy pulling the lever.
If someone says "ignore all the real world concerns", then does it even matter? It's the death of fantasy people, who cares?
But I would take an exception if it were family or friends. In a real scenario, saving them would be worth more than saving strangers or avoiding legal problems.
My reasoning is: say that bad city design leads to a lot of pedestrians about to be struck by cars in intersections. Does it really make sense to focus on the drivers?