You mean “How is silence an action rather than a non-action?”? Which I already answered before you even asked it?
When people ask you for help and you don't give it, that inaction is telling these people you won't give them help.
Inaction can make you be criminally accountable, so stop having this weird belief that inaction automatically clears you of all responsibility.