It doesn't assume that we can simply add their annoyance together. It assumes that, whatever function computes the badness of some badness applied to multiple people, it diverges and does so fast enough (where "fast enough" will probably include "exceptionally slowly", but in principle there are sufficiently slow divergent functions).
I don't think this is well enough established, though it seems plausible.