Being able to deal with people that make you very uncomfortable without creating a fuss is a fundamental requirement of adulthood. Regardless of the callous reaction of the un-invited person, it was not OK to un-invite them, at least for the reasons exposed in this article.
The organizer says:
> Being accused like that caused me harm.
of course it did. Just like the harm you did by cancelling invitations on behalf of a hypothetical discomfort of other people. It is exactly the same behavior.