I've always wanted arXiv to fill that role.

I understand though that simple anonymous comments don't cut it. It can devolve into youtube comments quickly -- personal attacks to the authors, low effort comments, spamming, "trolling", etc.

Those sites need a private reputation system: you can vote comments but the commentator is anonymous; it's hidden reputation should help sorting the comments.