There's no problem in letting anyone upload. The problem is in claiming that we should give the same amount of attention to the work of anyone, that "only what is said matters". Just like we don't run random code off github, we have no reason to read random papers on arxiv. And, even on github, anyone using a project knows that "who is maintaining this project" is a major decision factor.