If this happens, there's a lot to be learned from Stack Exchange on how to do this well. The stats.stackexchange.com site is actually pretty high quality IMHO.

Arxiv could use some of that reputation & moderation structure to regulate content. Cranks and time-wasters could be sidelined. It would certainly work much better than a typical newspaper comments section.

It would work by establishing expectations of how much substance should be in a response to a paper. Then getting the community to pitch in on the quality of responses. You still get echo-chamber effects with this, but the quality filter is generally worth it.