I believe the reason Arxiv is not trusted in some fields is that it's seen as a loophole that allows bad articles to avoid the filter of peer review and see the light of day. It would not be a problem if Arxiv was full of such articles, that would never cut it in an actual publication, but such articles are mixed with many that are eventually published and that just adds an amount of noise to the process that makes it hard to be sure what you're reading, especially if it doesn't exactly match your expertise.
In other fields, e.g. in machine learning, putting your paper on Arxiv is de rigeur although of course that means anonymous reviewing goes out the window, so there's legitimiate concerns there also.