Comment by ath0 | Hacker News Reader

Comment by ath0 | Hacker News Reader

I'm long past my academia phase, but recently led the PC for an industry conference (accept rate: ~15%).

1. Curation is important both for the physical limits (venues only fit a certain number of people), attention limits (attendees will usually retain only a handful of "nuggets" no matter how packed the agenda is) and interaction limits (you can't meet everyone at a large conference).

2. If the goal of a conference is not just to "stamp" research as somehow "approved", but to encourage discovery and knowledge exchange that deepens a specific area, it's important to apply that curation filter with an eye toward best advancing the goals of the conference. That means not just going for things that are okay, but those that best resonate with other presentations / attendees / research topics.

3. While the size of any one conference has to be fixed, tech has made it infinitely easier to create new conferences and journals with other focus areas. They may not start with the prestige of a larger journal, but if the papers published start to have an impact, it can catalyze an entire subfield of work.

Some conferences can be tied exclusively to "novelty" - ACM academic conferences - but others to "incremental advancements" - the bigger industry conferences in security, like Usenix Security and some to "best explaining ideas" - like Enigma.

There are new ways to find an audience for your work and create impact - that's part of the job now.

I'm well-known in a research community. I'm positioned such that I don't need more academic points. I've mostly stopped publishing in branded prestige academic venues, in part due to rejection rates.

My goal in doing work and writing papers is to see them disseminated. The acceptance/rejection process is asinine -- studies show it's basically random. I've had one paper in my whole career where the reviewers did a proper review (e.g. worked through the math). The rest were quick skims. Comments often show the reviewers never read the paper. The stuff that makes it through this process is often nonsense, while very high-quality work is often cut.

The very best paper I wrote in my career has never seen the light of day. It was shortened to a 4-page work-in-progress because a reviewer didn't read it (the feedback was literally nonsense: that the sample size was small enough to be anecdotal; I had the largest sample size in the history of the research field).

The only impact of this egoistical search for prestige-by-low-accept-rates is that people who have better things to do with their time leave, and that research dissemination is slowed.

Those excuses make little sense in the real world:

1) If your conference has a 10% accept rate, it's easy enough to book a bigger venue next year. I've been to conferences with dozens of people, and ones with tens of thousands. It all works well. Bigger ones work better, if anything.

2) PCs aren't thoughtful enough to do that well, and even so, the goal of a conference shouldn't be to select things which resonate with the entrenched PC. That's why many ideas need to wait for a generation of old, conservative professors to die to make it out there.

3) The whole obsession with prestige is stupid and misguided.

Journals and conferences ought to have quality bars. Are there typos and grammar errors? Were there clear IRB ethic violations? Did you use error bars on your plots? Was data fabricated? Is the research methodologically sound? Is it coherent and readable? And so on. If it passes those bars, it should be published. If no one reads it / attends a talk, that's okay too -- importance can and should be determined after-the-fact.

> Comments often show the reviewers never read the paper.

This. I was not in computer science, but in a different technical field, and this is sadly common. We would often have to appeal to the editor with "The topic the reviewer said we didn't address? It's in Section X. Get us another reviewer."

The biggest mystery in the whole thing is why someone who is volunteering to review papers anonymously would bother to do it badly when they could simply not do it at all.

Because they want to appear as if they are an active participant in the community.

No mystery. Behavior converges to incentives:

* You do get academic points for chairing a conference, and as a chair, you do need to find reviewers.

* A colleague is running a conference, and asks you to do a favor. You want to help your colleague. Reviewing papers wins you points with them, and declining to review burns bridges. When you're running a conference, you'd like them to reciprocate. Plus, they might be on a grant / hiring / etc. board / committee / etc. later on. Burning bridges in academia is very bad.

On the other hand, there is no incentive to invest more than 30-600 seconds per review. Neither you nor your friend really have any reason to care about the quality of the conference.

As this process repeats, people put in less and less time each time around, since it doesn't matter. The process converges to random noise.

>Reviewing papers wins you points with them, and declining to review burns bridges. When you're running a conference, you'd like them to reciprocate.

Surely they'd get upset if you rejected all of the good papers, thereby ensuring that they would have a bad conference.

Just accept people who has held a lot of conference talks before and it will be fine. That is the fastest and easiest way to review, so unless there is pressure to do things differently that is how most will do it.

If there is space still left at the end you can look at the others and take the first paper that looks fine until there are no spots left.

There is (for good reason) more focus today on diversity--broadly defined e.g. new speakers--for non-academic conferences these days. However, there were quite a few conferences in the tech sector historically that tended to have a core of "the usual suspects" with others grabbing a smaller number of leftover slots. TBH, I probably benefited from this over the years. (Conferences run by companies follow somewhat different rules but still usually have a stable of Top Rated Speakers who tend to get slots.)

Why would they care? They get their name as the chair of the conference on their CV. No one remembers who chaired a specific year or how a given year went. If they did, there's enough noise in the process it wouldn't be attributed to the review process in particular. Some years, weaker papers come in, and others, stronger ones do.

I've learned to address those ones diplomatically with "the topic you mention is now included in section X". Technically true, and lower friction.

Haha yes. Everyday example of this frustration (really happened):

"So when is their wedding?"

'Next week on Saturday.'

"Whoa whoa whoa, do you mean this coming Saturday, or the Saturday that happens next week?"

'Next week on Saturday.'

"Okay, gotcha, thanks, it was kinda unclear before."

I think this happens in all fields. It’s probably a professor on a PC dumping the review on an unsuspecting and overworked PhD student or MS student who really doesn’t care and just wants to get some sleep.

And yes - say what one may - PhD students are overworked and underpaid at least in most of the US.

I may have been radicalized during my short time in the academic world, but IMO, conferences are a really bad setting to disseminate new ideas. They just don't favor it. In practice, you have people preaching their ideas, a lot of people not listening, and a few misunderstanding. Nobody else.

Spreading ideas is better done on paper, with guided discussion, and without time limits. Or, in other worlds, on something like paper-split hierarchical internet forums.

Conferences can be useful to discuss and work over known ideas. For that, they should always bring papers that are already published, and had some community attention. The idea of debuting new ideas over unprepared people is antagonistic to that goal.

> My goal in doing work and writing papers is to see them disseminated. The acceptance/rejection process is asinine -- studies show it's basically random. I've had one paper in my whole career where the reviewers did a proper review (e.g. worked through the math). The rest were quick skims. Comments often show the reviewers never read the paper. The stuff that makes it through this process is often nonsense, while very high-quality work is often cut.

How come this is not fixed ?

Because the leaders are the people who made their careers in the current system and they wouldn't benefit from making things more meritocratic. These are the people who argues endlessly saying meritocracy is bad for reason X or reason Y, they just want to keep their current privileges.

I'm still a bit surprised how basic human defects can reach even the 'smartest' spheres of society.

> Comments often show the reviewers never read the paper.

And when they do, it's not sure they understood it or even put the slightest towards understanding. I've a rejected paper where one of the comment was that the header of a table featuring 4 columns named N, V, ADJ, ADV was "hard to understand". The table was between two paragraphs each mentioning nouns, verbs, adjectives and adverbs, in a paper mostly about dictionary...

studies show it's basically random

The "basically" is important though, because there are some nuances to it.

However, the point I've actually come here to make is that since publications are a strong factor for your career progress in academia, a corollary of the above is that making it in academia is basically random, too. Which is also true for other reasons, though: for every open professor position in a certain field, there are usually a number of candidates that are all equally highly qualified. But only one of them can get the gig. If the selection is not random, then it's typically based on other factors, such as, how well you are connected, your gender, whether some other professor at the faculty fears competition from you, etc. -- which may not be random, but is equally out of your control in all but a few cases.

My experience is that for elite schools -- Stanford and MIT -- the remaining factor is how much one is willing to cheat. There is a random component, a merit-based component, but most (and I have large n here) successful affiliated faculty candidates did so by cheating in some way.

That can be data baking, credit theft, or a whole slew of other techniques, but at least in my department, most new faculty at least at these two schools are in some way crooked.

Also, for nuance on random:

http://blog.mrtz.org/2014/12/15/the-nips-experiment.html

And one more: https://medium.com/@tnvijayk/potential-organized-fraud-in-ac...

From the article:

"99.99% of us are honest but the dishonest 0.01% can cause serious, repeated damage."

My experience is that this is much more like a 50/50 split at elite schools, at least when you look at people who succeed at making it to faculty positions. BMJ estimates 20% of publications are based on fabricated data:

https://blogs.bmj.com/bmj/2021/07/05/time-to-assume-that-hea...

That sounds about right for what I've seen at MIT. Note that 20% of publications being based on fabricated data is in-line with my 50/50 split figure. Researchers who cheat only do it part of the time, and often in ways which don't involve direct data fabrication. Critically, the numbers go up significantly for high-impact publications -- they types that make the news and make scientific careers. By the time MIT's PR machine picks up a publication, and the press picks up from there, the odds of it being fraudulent are much higher than 50/50.

> 3. While the size of any one conference has to be fixed, tech has made it infinitely easier to create new conferences and journals with other focus areas. They may not start with the prestige of a larger journal, but if the papers published start to have an impact, it can catalyze an entire subfield of work.

Does it though? The largest conferences I go to as a CS academic have hundreds of people. There are academic areas where 10x people participate. The size limitation is a self-imposed excuse to keep acceptance low. I have been PC chair of two conferences and my attempts to expand the conference numbers were shot down by the steering committee precisely for this reason, not because we couldn't find a larger room.

Try "publishing" in https://researchers.one

Reply on news.ycombinator.com