Think about it: The number of inaction's you have is infinite - there are an infinite number of different things you could have done that would have saved someone's life. So an inaction is the default state in this world, but taking an action: That's something that has meaning, and people chose not to kill by taking an action.
Aside from the above, you are assuming that 5 people is worth more than 1, but that's not something you can know, you can't know the worth of another person. So who are you to choose to kill someone?
Mathematically the expected outcomes should be the same, but morally they are not.
In real life you never get perfect information, so doing nothing as opposed to pulling represents the humans uncertainty of the parameters in the scenario.
“First do no harm”.