6 karma · joined May 10, 2022
If I'm looking at the right document, one question was about which city out of Chicago, New York, and LA have the greatest annual temperature range (accompanied with a plot). Almost all respondents said New York or Chicago, rather than LA or "All equal".
It looks like they used multiple choice quizzes to determine both knowledge in science and a propensity to respond "don't know" indicating confidence. Any "don't know" response was counted as an incorrect response, while a correct guess increased the participants "science knowledge".
Thus, a willingness to guess something at random in the multiple choice test would both increase "science knowledge" as well as make the participant appear overconfident.
For all of its issues, I still think there would be some value in encouraging people to find issues in papers and publicly note them instead of relying on an oral history of which papers are accurate (hence my suggestion to encourage people to reproduce or disprove papers, while slightly penalizing submitting new research). Many fields have reproducibility issues, and my experience has been that research groups informally build their own internal lists of (seemingly) inaccurate publications.
It would be naive to believe that even the most devoted individual would opt to starve rather than pivot their work to eat. As it stands, there isn't much an incentive other than "truth" or general political reputation management to document the reproducibility of other work. Erroneous works make it difficult to find relevant, accurate ones.
Maybe a digital currency that's created by reproducing (or disproving) existing papers (weighted by lack of reproduction) and with which the author of a submitted manuscript "pays" reviewers and editors, which they in turn use to submit their own papers?