Moral choice is not always the right choice. The fact that the choice is right or logical doesn't make it moral.
> Surely inaction is just as morally unjust as action?
Nothing sure about that. For example in case of "trolley problem" I would not reroute the trolley so that 1 person dies instead of five because I believe that saving people is optional, but not killing them is mandatory. Or I would reroute the trolley accepting that even though it might be correct it was immoral.
> In more direct terms I, for one, would feel no guilt killing to save my own life from an aggressor, or to save those of my family/friends.
I would feel the guilt if I killed someone.