Deep impact: unintended consequences of journal rank

--- While they do comment that retractions are just the most extreme version of this problem, the retraction rate of .02% they mention near the beginning does not strike me as a crisis in science.

--- I'd also like to see more acknowledgment that the degree of this kind of problem depends pretty heavily on the field instead of just lumping everything together as "science." Physics has had its share of frauds (e.g J. H. Schon[1]) and certainly puts some weak papers in big journals, but I have found that publication in one of the big journals really is meaningfully correlated to quality on average.

--- Earlier sections of the paper criticize the higher unreliability and lesser quality of papers in high impact journals. Much further on, there's a paragraph or two acknowledging that these papers, by dint of being in these journals, typically have more novel results and receive way more scrutiny than typical papers. Thus it would be unsurprising if the true error rate actually were higher, and certainly one would expect the error detection rate to be much higher for these articles. Further, I expect this meta-result (big papers come up short more) would persist even if you got rid of the journal model like they propose.

--- That last bit is something I don't feel they address at all. Any model of "filter, sort and discovery" is going to produce papers which are more successful/visible than others. The career incentives of scientists will still be essentially the same in a new system. Setting aside the path-dependence problems of switching to a new model (when basically all of the present actors are well calibrated to the old system), I'm very skeptical that their proposed solutions will actually prove less gameable than the current one.

[1] This is a pretty good book on the subject: http://en.wikipedia.org/wiki/Plastic_Fantastic

If you want to be a tenured professor, you'd usually be expected to have at least a certain number of publications in the top journals.

Also, I remember an interesting comparison amongst the Top 10 Econ departments, that compared grad students and the amount of top-journal publications they had 5 years after graduation.

The bottom line was that the Top 5 students in MIT\Harvard really have much more publications than any other grad student. But, the median Harvard\MIT one doesn't perform better than any grad from tier 1 school

The title for the typical paper in medicine should be "we didn't use a large enough sample size"