Our broken peer review system, in one saga
familyinequality.wordpress.com
familyinequality.wordpress.com
LOL - so essentially: "printing/copying of PDF is cheap, so Y U no publish this 'study' of a regression that a first-year stats student could do with an interpretation of that regression that is devoid of any scientific method?"
Yes, if your paper is weak to begin with, you need to jump through more hoops (i.e., sell your research better).
Look, getting your papers published sucks sometimes, but if anything, the OP confirms that peer review works. Or maybe what it shows is how it fails in that when people take weak research and just keep submitting to journal after journal until one is tricked into accepting.
(the further I read on the page, the more I'm convinced we're subtly being trolled here: "The new knowledge was all created two years before it was published." "new knowledge"? A regression on a publicly available data set with the main dependent being an ambiguous 3-option question on a highly contentious and nuanced topic; and some 'discussion' which the authors admit in the first paragraph is unverifiable and basically an opinion piece?)
(edit: typo)
The suggestion in the linked article that, "hey, we ran a regression so now we can just come up with whatever interpretation we like" irked me as well, but you really do have to read the comments of the peer reviewers: one is peeved by the fact that they don't talk about his/her pet topic, another that they don't include this or that kind of historical context, another tries to be smart about statistical methodology but in the process shows their incompetence (a beta coefficient is a measure of effect size, you dummy)... when all the authors want to do is to point at the fact that opposition to pornography has declined over time. You can argue that this is not interesting, that this is obvious, but the whole point of science is to not accept things just because they seem obvious.
A new journal launched recently, by the way, which is aimed at exactly this sort of finding: https://sciencematters.io The idea is that scientists will be able to publish empirical results without the requirement to craft an entire story around it. Science needs theory but it also needs facts.
It’s hard for reviewers to politely articulate “we just didn’t think your paper was cool enough for our journal” or “we thought your analysis was weak”, but most of them did seem to be trying. My impression is that the author didn’t really take the criticism to heart, and had trouble reading between the lines.
These journals surely get many similar paper submissions, and this authors’ opinion notwithstanding, they can’t publish them all.
I agree that the system is too slow, too arbitrary, and too high-stakes though. It’s tragic that quantity and venue of published papers is the main way academics are judged.
It would indeed be nice if scholars could (as an option) publish their papers without as much oversight, and then have more of the discussion back and forth about the paper’s relevance take place in public. Then if the paper turns out to be significant, it could be republished in a leading journal.
To go back to basics: start with a theory or model, make a prediction based on that model, then test to see if that prediction is correct. In this work, the authors explicitly stay away from anything other than a hand-waving explanation of their results, and seem surprised that the reviewers keep asking them to "frame" their results in a broader context.
It would be as if Newton published an observation of fruit falling off a tree, and failed to mention gravity. An interesting observation, perhaps, but not an advancement of science.
If you put a paper there, and someone reads it/finds it useful, or if someone does a literature search on the subject, the arXiv entry will get cited.
<looks at arXiv, notes lack of a sociology section>
I don't know why arXiv hasn't broadened its scope outside of the natural sciences, but I'm certain that a) Ginsparg has a good reason, and b) they're happy to help others implement similar systems.
To be fair, most of these non-traditional systems cater almost exclusively to hard sciences and don't focus on sociology.
The same basic problem occurs in all fields in that many reviewers and editors are basically idiots. When you get a rejection you look through the reviewers comments for the useful suggestions (mostly these are about spelling or grammar, but occasionally you will get a real gem) and send it out again. Whatever you do do not change the paper to suit the reviewers or editor. Just shoot it out again as quickly as possible.
I have had the situation where all three reviewers said publish and the editor rejected the paper - of course this paper has gone on to get over 600 citations so it was not the best decision that editor has ever made.