So we end up coming full circle from "here are the rules" to "play each situation by ear". Ethics is just too dang hard!
It would be a much less frustrating world if the robber who takes your money would just tell you "hi, I'm taking your money because I can and because I like it" instead of earnestly believing that he deserves it for some complicated reason (pick your example).
Most muggers don't try to justify themselves philosophically, thankfully. That would just make the experience even more painful.