Quality, not quantity: ending science's "academic prostitution"
arstechnica.com
arstechnica.com
It would be very nice to see a distributed publication system taken up by the scientific community, based on circles of trust where anyone could partake in the peer review process.
I understand that when double blind reviewing was added to the area of AI research, the distribution of who got things published changed dramatically. I don't see how a circle of trust will help with that. The other part of the problem is getting people who are willing to do thorough reviews.
Time is money, and with the massive pressures to publish, it has become somewhat common in CS for professors to hand off reviews to students who may not be qualified. Or alternately, reviews can be too short or not thorough enough. This is really illustrated well in conferences, which ironically are often valued more than journals. The extreme number of submissions coupled with a very limited assessment time and strict page limits leads to some horrible reviews that help no one. Even bad reviews should be useful in the sense that they should point out what is wrong or how to improve the work.
It isn't possible. Much better to have multiple intersecting circles of trust than to pretend that there can be one unbiased judgment of a work.
And when I say "isn't possible", I mean, "isn't possible". At all. You can't do it. I could all but prove this mathematically with enough time, using terminology and concepts from the machine learning bunch. (Their definition of "bias" is one of the most profound things I learned in my entire 7-year college education.)
Double-blind reviews can still be done in the circle-of-trust regimes. Obviously there's a point where the work is unblinded and you have to deal with that, but it hardly strikes me as impossible.
In some cases, if it's an author I already know of, I don't even bother looking to see where it was published until I actually need to write out a bibtex entry to cite it. Doing things that way, I've cited papers from publication venues I've never even heard of, so they aren't really doing any sort of information-filtering work for me. In those cases, the article may as well have been a tech report for all it mattered to me, since the publication venue was really serving as nothing (from my perspective) except a fancy place to deposit PDFs. Admittedly, that works mainly when I already know something about the author (or someone I know can tell me something about the author). If they're a total unknown, I'd probably be more wary of citing stuff that didn't already get into a venue I respect.
While I agree there is more to science output than publications, candidates for post-doc/faculty positions shouldn't be judged on "effort", but on making real contributions that bring progress to their field of research. Judging that on anything else than quality publications becomes difficult, but I look forward to what people may do about it.
So far I've heard mostly talk and little action.
I think people actually are already pretty good at judging research quality on something like "effort", or more to the point: how good a scientist is so-and-so? But it requires familiarity with the person over some time, and tends not to scale well. If you ask a scientist about the quality of research of one of their colleagues they've been in the same department with for 10 years, they'll rarely refer to publications, and they often won't even refer to discoveries. Instead, it's inputs rather than outputs to the research process that they evaluate: how well is the person managing his/her lab, how insightful is he/she in picking problems and approaches, how good at mentoring students, how well does he/she really understand where this field is and where it's going / should be going, etc.
There is correlation, of course, in that the good scientists most often also produce good results. But at least among the researchers I know, if you have to pick one as the "real" one, inputs rather than outputs are it. If you observe someone for a while, you can tell whether they're a good scientist or not, and you can also tell the places where it doesn't correlate with outputs--- someone's been doing everything exactly right on an important problem but just hasn't gotten a break (or maybe did get it, but it came a month after they were scooped by another group), or someone else is kind of mediocre but got lucky with a big result. Neither's the common case, but it's not that rare either; it's not like research is just a crank you turn and results come out. =]
I guess the main issue is how to scale it. I think the above is pretty much how people already do make decisions when they're familiar with someone's work. If someone is up for tenure, my impression is that profs in that person's area will rarely even bother looking at publication lists when deciding whether to vote yes/no: after working with the person for 6 years they already have a good idea of whether the person is a good researcher they want to keep. It's mainly the people in other areas (in a large department), or in administrative roles, who seem to have no choice but to use the CV as the main basis of their tenure decision.