Secondly, it doesn't fix stupidity. A participant who earnestly takes the quality goals of the system to heart instead of focusing on maximizing their take (thus, obviously stupid) will still make bad classifications due to that reason.
1. I would expect any paid arrangement to include a quality-control mechanism. With the possible exception of if it was designed from scratch by complete ignoramuses.
2. Do you have a proposal for a better incentive?
2. Criticism of a method does not require that there is a viable alternative. Perhaps the better idea is just to not incentivize people to do tasks they are not qualified for
Agreed, and would add that it doesn’t fix other things like lack of skill, focus, time, etc.
An example is the output of the Amazon Turk “Sheep Market” experiment:
https://docubase.mit.edu/project/the-sheep-market/
Some of those sheep were really ba-aaa-ad.
edit: ugh. it's even worse, lmarena itself is a proprietary system, so the users presumably don't even get the benefit of an open dataset out of all this