As for open review ... You also don't strike me as someone who has reviewed a lot of papers.
I've served on several program committees, and reviewed hundreds of papers. I'm happy with blind reviews.
You're judging by reputation rather than substance. That opens up the possibility of manipulating reputation to achieve career goals instead of doing the hard work.
Is there an alternative, though? When hiring someone we use their degrees, experience and credentials, all of which are indirect ways of estimating competence. We cannot realistically expect people to know all about someone’s career when interviewing them, it’s not realistic.
It’s the same thing for careers evolution. HR people don’t have time or skills to know about the science, and good scientists already spend way too much time reviewing articles or grant proposals, and sitting in various boards and panels. There is no need for more administrative work.
I don’t think the current systems are ideal, but I think improving it would by fixing incentives would be more efficient (and that’s not only for scientists, it’s the same for any major organisation or company).
> Is there an alternative, though?
I said "read the god damn papers" but this is a condensed version of saying be more nuanced and familiar with the person's work. Metrics are signals, not answers. They are always fuzzy and we need to recognize that and bring this into the core of how we evaluate. You'll never be perfect and while we know to not let perfection stand in the way of good enough neither should you let lack of global optima stand in the way of progress. There's always room to improve.
At the heart of academia we push human knowledge forward, even if just by little nudges at a time. But this means the environment changes as we succeed. What worked in the past may not work moving forward. And as we all should be intimately familiar with, given enough time Goodhart's Law always wins.
But there's so many problems with becoming over reliant upon metrics. It's not a static measurement, it is a degrading one. This is because we get lost seeking success for ourselves because doing so makes it easier to do the work we want to (and ideally more impact on humanity). Because of this the best way to advance your career is not to continue researching but to grow the size of your lab and get grad students and post docs to research. You need to become a PI, who's job is political and about funding. This person does need a strong PhD background but it's also silly that it's extremely hard to find a path where your primary job after getting a PhD is... Research...
But pushing the edge of what we know is fucking hard. It should be no surprise that it is hard to evaluate the performance of people trying to do this. So forgive me if I think in addition to using metrics like citations, venue pubs, h-indexes, you can take the time to read you colleagues papers. I'm really not asking for much here.
One of the things I also find rotten in academia is this isolation. It's weird that we don't know, intimately, the work of those in our department. We should be reading their papers in the first place! I'm glad academia encouraged collaboration with groups outside your institution, but it shouldn't come at the cost of collaboration in house. Both are valuable.
There's just so much more and I think it's why many are willing to tear down the system and start over. I'd rather not, but things are too broken for patchwork. We don't need to tear it down, but we do need to do some serious renovation.
I encourage everyone to think deeply about the problem, even if you don't think there is one (especially if you don't!). Address the underlying assumptions that no one says. Consider when they hold and when they don't. Then ask if this works. If you believe so, great! I'd love to hear that opinion. But I find it absurd we can't spend a little time questioning the vehicle we are all riding in. It should be questioned frequently and regularly. Because maintenance is far easier than fixing something when it's broken.
In practice, you reject the vast majority of the candidates by reputation, publication venues, paper counts, grants and awards, and similar metrics. And then, if you are particularly interested, you may devote a few days on superficial reading of the works of the shortlisted candidates.
> Hard work doesn't scale
It's true that there are hundreds of candidates but that's... Scale. Getting a PhD is no easy task. And I'm not saying you should abandon the metrics all together, that would be a naive misinterpretation especially given what I made explicit. Instead consider the effort in evaluation should scale with the role. Maybe you can reject 80% of candidates by citations and indices alone. Do you miss some gold candidates? Probably, but it's fuzzy. Your remaining 20% you can consider their niches, funding capacity, and read at least the abstracts of their top works. But in your final selection you should be nuanced and read a selected list of their works (not all. Let the candidate tell you).
Is it hard? Hell yeah. But why should something being hard stop us from doing it? That's called giving up. It's a crazy mentality to have as a researcher.
> you may devote a few days on superficial reading of the works of the shortlisted candidates.
Come on, I can't be the only one that thinks this is ludicrous.
And even when your work is evaluated, it's based on superficial reading. The people reading it (who are hopefully experts in the subfield) don't have the time to become experts in that specific niche to judge it properly. What they do is similar to peer review. And like all peer review, it's far more useful for spotting weak points and suggesting improvements than for ranking people/works.
> Evaluation by reputation and metrics is the norm. Evaluation by the quality of your work is the exception
> And even when your work is evaluated, it's based on superficial reading. The people reading it
This is exactly what I'm complaining about and saying we should do better.
Just because things suck doesn't mean you need to defend them. We can in fact change things.
We're researchers. We solve crazy hard problems all the time. I'm not saying our "house" needs to be completely in order, but right now it seems to be in dismay. I'm certain we can solve this. We put a fucking man on the moon, we found the Higgs and gravitational waves, we made computers talk, and we rid the world of deadly diseases. Surely we can solve a problem like this. If anything, those should be reasons to motivate us to solve this, so we can do more! But I do think a problem is how people interpret economics. Maybe a research paper doesn't directly lead to a trillion dollar company, but those companies certainly rely on a lot of publicly funded research. The returns on investment are pretty high and that's worthwhile.
[0] I believe some feel the need to defend a system due to their success in it. This needs to be disentangled. A critique on the system is not a critique on "you" or "your work".
Of course you're happy as the reviewer. I would be too if I could criticize completely anonymously and have it counted. Me as the reviewee? Not so much.
I always thought it was funny: in the US, it's a fundamental right that you should not be criticized anonymously (6th amendment). Ostensibly this is to prevent laundered dictatorial rule. And in academia it's the norm lol. Makes you wonder doesn't it.
I would support double blind reviews in both roles. I don’t think it’s fair to evaluate a manuscript on the authors line up. I think it’s irrelevant and encourages arse-kissing to get big names on papers and reviewers that are too deferential or combative just because of their personal history with one of the authors. I don’t think it is healthy or productive, even when I am one of the authors.
What is your perspective, why would you not like it?
> What is your perspective, why would you not like it?
Authors should be hidden from reviewers. Reviewers should be made known to authors. Why? When some reviewer dings me because he/she doesn't understand a fundamental concept, or some technical detail, or because I didn't cite their work, I should be able to have some/any recourse. Right now these people are completely shielded by anonymity.
Jurors are anonymous when it’s needed, for the same reasons voting is secret: to avoid coercion, blackmail, and retribution. We should want to keep reviewers anonymous for the same reasons. Prosecutors and judges are not, of course. And I don’t need to ask myself, all of this is obvious.
> Authors should be hidden from reviewers.
So we…half agree? As I said, think it would be much healthier to have double blind reviews, because what’s being reviewed is the work and not the authors.
> Why? When some reviewer dings me because he/she doesn't understand a fundamental concept, or some technical detail, or because I didn't cite their work, I should be able to have some/any recourse. Right now these people are completely shielded by anonymity.
And for good reasons. When I reviewed a paper from a big lab as a post-doc, it would have been impossible to do it properly if I had to fear the professor’s retaliation. Anonymity is a shield to avoid direct confrontation.
Rogue or stubborn reviewers are why we need more than one of them and editor discretion. The editors are not stupid, they know these patterns.
Review works well at small scales because people are more passionate about the work than the money or career, though those matter. There's better accountability in smaller communities because they're more reliant. But blind is often a farce there since you recognize one anothers writings, subtopics, niche, whatever.
At scale review sucks because there is no accountability. Venues need to reject on percentage of submissions, not on the quality of work that year. It's insane it's a zero sum game. And at scale everyone is over burdened and quality control goes down, so malicious/lazy reviewers are even incentivized since they gain time for being lazy, they have less scrutiny when rejecting, and they increase the likelihood of their work being admitted when doing do (I mean every work can be legitimately criticized, so a reject is ALWAYS defendable, even if seen as harsh. This isn't true for accepts). But also at scale, blind is weird. It exists for small labs but not for big ones.
So I don't know. I think many forgot the point of review. That we should all be on the same team (team science), and competition should be friendly. That reviews should leave the authors feeling like they know what they can improve on not like a review read a different paper or just read one figure (and wrong), and just dismissed (that's not a review). And that wastes lots of time and money as well as increases the problem as papers get recycled through the slot machine.
So I don't know. But I do know the point of publishing is to communicate with peers. And this can be done by open punishing. Sure, this might make the signal of citations and number of papers noisier, but I think it's already pretty noisy even if we don't want to admit it. And maybe it's not a bad thing for those to be noisy. I think we rely too heavily on them currently. Maybe if it was clearer that noise existed we could be more nuanced in our evaluations, which is what's frequently lacking these days. Who knows. But I do know the system today is not one I want to continue being in and this sentiment is far from uncommon. So something needs to happen.
The academy is so corrupt it should be "shattered into a million pieces and scattered into the wind". H Index ego, P hacking, lazy lies because "no one tries to reproduce", citation cartels, institutionalized systemic academic dishonesty/plagiarism at the highest levels (presidents of top universities and likely their underlings), "reviewer #2", etc. It's all bunk, and most of it is designed to take bright-eyed Ph.D's and grind their soul to bits before they can realize that they can both out-earn and out-produce (in an objective sense) most of their professors by simply leaving the academy.
Being ineligible for tenure should be a badge of honor, as the whole system should be abolished and the world would be far better for it - particularly, science would advance far faster.
Far more of HN needs to read Paul Feyerabend and actually internalize what "epistemological anarchism" means. The academy does not hold any particularly valid claim to being closer to observing "objective truth" than anyone else. The allegory of the cave/divided line (the primary justifications for a lot of academic wankery) was one of the original lies made by a nascent and authoritarian philosopher class. We should have abandoned these ideas long ago.
Oh, and if you leave the STEM fields, the academy goes into full on "charlatan" mode. I go into most social sciences and find a wasteland of footnotes on Foucault and Lacan. That Situationalist International was so successful that the academy is still choked by an obsession with french fashionable nonsense is an indictment of the whole of the academy in the west. It's all intellectually bankrupt.
> You're already familiar with most of them
> I've served on several program committees
You speak with an air of authority and that we should trust you but give no actual reason why. Use your experience and expertise to educate us. Don't just say I'm wrong, say why I'm wrong. Convince me you're right not through ethos, but logos. This is what everyone hates about review 2. Don't become him.
I believe you've also missed the substance of what I originally suggested. You're so caught up in the standard way you injected it into my comment without even noticing. You forgot the point all of this. It to climb the ladder of academia, get grants, and all that. It is about advancing science and moving humanity forward. The way to measure this isn't meaningful at our disposal. You can't measure this by citations or indices, by publications or venues. Again, they can be a signal (I'm not saying to throw out the baby with the bath water), but I don't believe you're naive enough to not be aware of how frequent progress is made by those who fail on those measures, how frequent it comes from a dark horse. This should be enough to suggest that there's more to it than what we usually consider. Academia has always been about challenging the status quo, even if it is just to verify. The problem with academia is it's created to give refuge to those who want to spend their lives dreaming but has become so structured you can only dream a certain way. I get why. It's hard to differentiate useless pipe dreams from those that push us forward. But in creating the structure to prevent people from taking a free ride you created the monster you tried to stop. Only now it has more power.
So don't just wave your hands dismissing, elaborate. Be academic not arrogant.
> In what way? Their prestige?
In the sense that I've read most of their papers, I've even reviewed several of them.
I've literally just handed in a report for an expert committee to hire a new faculty member, and I had read papers from all applicants that were relevant and reviewed for more than half of them.
>> I've served on several program committees
> Then maybe stop complaining and start critiquing.
I'm not complaining about anything. I just tried to lay out my point of view.
And yes, I'm aware of the painful measures that citations and impact factors are, but, like any other HN reader, I'm aware of Goodhart's law.
My point is that we're (I'm) not using citations or publication points to rank researchers. We know them.
> In the sense that I've read most of their papers, I've even reviewed several of them.
So what was the point of your comments