Hacker News Reader

Comment by Fomite | Hacker News Reader

This. The biggest and most meaningful way to do this would be for people sitting on search committees and tenure/promotion committees to say publicly that candidates wouldn't be punished for deliberately publishing in OA journals.

View on HN

The problem is the following: if there is no established OA journal in the field, you can't distinguish the candidate who "deliberately" publishes good quality work in a non-selective OA journal, from the candidate who publishes low-quality work in the same non-selective OA journal.

The only way to tell is to read the papers and judge for yourself, and this is the root of the problem: evaluating candidates based on the number of papers and the impact factor (or some analogous metric) of the venues is much, much easier for committees, than actually reading the papers.

In a very rough analogy, this is why we have film critics, because we can't trust other institutions in the industry (studios, individual theaters, directors) as a guarantee of quality.

If a panel of respected tenured professors in the field ranked the best works in open access journals each month, or if there was a high profile yearly prize for best open access papers of the year, that could help disaggregate reputational effects from the publication itself.

Bootstrapping this sort of thing is hard. I'm sure Nobels didn't signal as strongly in year one as they do now.

Exactly, this nonsense of mostly just counting publications must stop. If at all, there is only a very light correlation (or perhaps even a negative one, who knows?) between the number of publication and the quality of a scientist's research. And what's so crazy about it is that everybody knows this. No one has ever been called a great scientist, because he published many articles. Such scientists may be called prolific, but that doesn't necessarily reflect on the quality of the content.

The same for counting quotations, by the way. This may be different in STEM fields, but I'm in the humanities. In the humanities bad papers are sometimes quoted as often as brilliant papers, because everybody wants to correct the authors. I've seen more than one colleague literally make his career by writing a polemic and obviously wrong book (books=often improper peer review) and then sneaking a few apologetic articles in which they half-retracted their claims past the reviewers. What's worse, I know postdocs who are actively looking for such untenable topics that will surely stir up a reaction. We're talking about theses, conjectures or claims that nobody defended previously, because they are so obviously wrong or flawed. I've also met people at the postdoc level who literally replied "Who cares, it's one more publication" when I raised doubts about their main claim in an article and mentioned that it might not 'work' for purely formal reasons (but adding that I'd have to check, to be polite).

I'm sure it's a bit less drastic in CS and the STEM fields, but in the humanities publication counting almost enforces mediocrity and the publication of uninteresting, pointless papers with vastly overlapping topics - and it's completely obvious that it does so.

The situation is particularly frustrating, because the system makes it harder to publish good science. Good science needs larger studies and/or better formal underpinnings, both of which take more time and more effort than duplicating fake papers about fake problems.

Well, enough ranting for today, you get the point.

It's almost as if academia were set up to advance the careers of those within academia rather than to advance human knowledge.

People will game the metric you impose on them.

The onus is on the administrations to get a metric that is gamed by advancing human knowledge, instead of throwing logs at flame wars. That's why they get the big salaries.

Nobody punishes you for publishing in OA journals, what matters is the IF and JCR rank of the journal, not if it's OA or not.

If you find an OA journal with cachet matching Nature or Science, I would be happy to submit stuff there. As far as I know, there's nothing even particularly close.

In other words, I doubt anyone is explicitly penalized for submitting to PLoS Biology instead of Journal of Neuroscience. However, there is an implicit penalty for not submitting to Nature/Science/Cell.

[As an aside, I really hate the obsession with IF and all these other "quantitative" measures of research quality. They're all incredibly ad-hoc, someone easily game-able, and yet otherwise-smart people take them so seriously.]

Yes, they are pointless and arbitrary metrics.

Competition for positions in academia has gotten so bad that it doesn't matter. Like Google rejecting perfectly good candidates just because they can, but in Academia even south central Kansastucky state university has enough excellent applicants that arbitrary metrics do more good than harm.

At least in my country, I'm thankful for these quantitative measurements, that difficult corrupt practices: Oh, I assure you he's a great researcher, just not in any measurable way. He's my nephew you say? Well, that's not the point...

Couldn't you just require people to recuse themselves from hiring decisions involving relatives?

Particularly for early career people, I suspect most of these "quantitative" measurements are basically reflecting where and wtih whom you worked.

There's also less nepotistic problems. For example, I've moved between different sides of the same overall field several times. That means, for any given reader of my CV, only about half the "impressive" papers read to them as impressive using most subjective measures.

> Couldn't you just require people to recuse themselves from hiring decisions involving relatives?

They used to have this nice trick where you hired the son of your friend and your friend hired your son. But at least now there are some non-subjective requirements to be met before anyone can get admitted.

While this is important, in many fields their are cliques of related researchers (1) on the grants committees and (2) on the journal committees. These people know who works on which problems, so blind review is a farce.

Also, the concept of IF being a proxy for rigorous review and work quality is weak at best. Just read a few pages of http://retractionwatch.com/

This seems like it would be fixable with a fairly small amount of money, relatively speaking: offer something like $250K for the most important papers that would otherwise be published in Nature to instead be published in an open-access journal. After all of the major papers for several months are published in open access journals and the most well-known journals are reduced to publishing only the dregs that are left, opinion could change quite quickly. Sustain this type of payment for a couple of years and I'd bet there would be a sizable number of other people publishing open-access, since to be published in Nature would imply that your paper wan't important enough to be offered the $250K reward.

(Disclaimer: I used to work in this field at biomedcentral.com)

Or just make Science OA. Unlike Nature and Cell, it's published by a non-profit (the American Association for the Advancement of Science). In principle, the American government could just give them money to make everything freely accessible. Or the publishers could ask authors to pay publication fees with waivers for those unable to pay, like most OA journals. Since the goal of the AAAS is (at least theoretically) to publish papers and not to make a profit, the fees need not be much higher than other OA journals. Given the average Science paper probably costs something like $1 million to produce, I doubt the authors would balk at paying a few thousand to make their work accessible to everyone.

If you had any kind of realistic measure to identify "important papers that would otherwise be published in Nature" then the problem would be already solved, since all the evaluation criteria could look at that instead of the journal ratings. However, we do not - currently the only actually usable way to identify "important papers that would otherwise be published in Nature" is simply asking "has this research been published in Nature?"

We don't necessarily need OA. Nature allows you to publish your papers on the arxiv. If all papers are on the arxiv, and with the journal, the negotiation position of the journals goes away.

Impact factors and similar metrics are part of the problem. These are metrics that support flashy research, not good research. If anything the Impact Factor is a sign of sloppy research (there's a lot of evidence for that, e.g. the higher the IF the more likely there's a retraction).

But it's not an easy problem. How do you grade from 0 to 10 good research from a candidate?

there's a lot of evidence for that, e.g. the higher the IF the more likely there's a retraction

The higher the IF, the more likely it is the researcher trying to do something 'hard' and the more people will scrutinise the results. These two factors alone will lead to more retractions.

Someone publishing something completely uninteresting in a journal no one reads will never be in a position to have to retract their paper, no matter how flawed, because no one will notice and no one will care.

But given the IF and ranks of OA journals, that's effectively the same thing. What you'd have to do is essentially give OA journals a handicap.

Not to mention the opportunity cost associated with OA fees, which can add up if you're talking about lab startup.

Reply on news.ycombinator.com