Hacker News Reader

Comment by hueving | Hacker News Reader

>Since conference proceedings are typically ignored researchers generally won't put their best effort into these papers.

It's interesting, because this is completely the opposite in many CS fields (e.g. security). The top conferences carry much more weight than any journal. It's probably related to how fast the field changes.

View on HN

HCI and graphics are other examples: a SIGCHI proceedings paper carries way more weight than a paper in one the HCI journals. And a SIGGRAPH proceedings paper is better than than a paper in something like Journal of Computer Graphics Techniques, although they've recently converted the SIGGRAPH proceedings into a special issue of ACM Transactions on Graphics (which is now more prestigious than other issues of that journal, because it's really a proceedings volume), mostly to avoid the confusion from other fields that don't consider conferences to be publications. Those two conferences actually have nearly exclusive prestige in their field, to the extent that they're the only top publication venue: if you're working in HCI or graphics you more or less need SIGCHI or SIGGRAPH publications to be considered a legit active researcher. They also tend to be more selective in peer-review than the journals are, with acceptance rates in the 10-15% range, while journals are usually more like 20-35%.

In AI I think it's a bit mixed, and maybe a US—European difference. Europeans tend to value journal publications in places like JAIR, JMLR, various IEEE Transactions, etc. more. Americans also value the best of those journals, but I get the impression that conference publications at places like NIPS, AAAI, and ICML are at least as good, and definitely better than a lower-tier journal.

One day when I was doing undergrad research, I helped one of my professors move to their new office[1]. He had an entire bookshelf of SIGGRAPH proceedings going back over 20 years.

[1] Before anyone brings up "undergrad research slave labor", I'll have you know I received a copy of "Gödel, Escher, Bach" for my troubles.

I was a paid undergrad security researcher that made double that any of other on-campus student job, where labor was cheap and abundant.

I'm not in CS, but I've heard of this. It's definitely aberrant. Some other fields change just as rapidly so I'd just chalk this up as a cultural difference.

In many CS fields, workshops are the equivalent of conferences in other fields such as physics. It is important to judge each field on how the primary peer-review publication is done, not by some abstract rule that is supposed to be the only valid standard.

As a side note, I've heard people from the humanities scoff at physicists and their belief that journal papers are worth mentioning. The only real publications are books.

Actually, there are workshops in many CS fields which are selective (not as much as big conferences, that's for sure) and as serious as conferences or journal on peer-reviewing.

The name "workshop" mainly means that the focus of the event is very specific and that the attendance is intended to be small so that real discussion with everyone can happen (contrary to big conferences where there are hundreds of people).

For example my last published paper was submitted to a workshop [1] and I had four different detailed reviews, which had nothing to envy to big conferences reviews I could have got for it. This is an example of a not very selective (because it has little submissions) workshop but which is very serious and would most certainly reject any computer generated paper.

Another example could be CHES [2] which is a workshop which grew big and is now the most prestigious conference (again, that means more prestigious than journals as in most CS fields) of its field (crypto hardware/side-channel security). In this case the name "workshop" sticks, but it is actually a full-fledged conference.

Yet another example of a small workshop [3] (still in the same field) which does serious peer reviewing (I had three looong and detailed reviews when I submitted my paper there). Last year they selected 8 papers for presentation the day of the workshop, and 4 of them were selected by a journal [4] for subsequent publication of an extended version (and after another small round of reviews since there were already the reviews of the workshop which were deemed serious enough by the journal's editors).

Maybe this is particular to the field of security (could it be due to the hacker subculture background it has in addition to its academic background?), but I guess my point is that it is not possible to generalize how the publication system works for all sciences, let alone for all academic research.

[1] http://pprew.org/

[2] http://www.chesworkshop.org/

[3] http://www.proofs-workshop.org/

[4] http://link.springer.com/journal/13389

Sorry, I was too general. I am very much aware that there are selective workshops in CS (I've been to some). I think I mostly wanted to point out some differences. My main wish was to convey that publishing venues are different, and the name of the venue is not always an indication of the type of publishing.

Thanks for pointing out that I made the mistake I was trying to correct.

Actually this is what I understood from your previous post. I was trying to strengthen your point with concrete examples and thought the best way to do it was in a reply. Sorry if that was unclear!

No problem. It was one of my pet peeves when others (typically physicists) devalued the work of the computer scientists at the university I was at just because they believed that journals were the only acceptable form of publishing.

My read on it is that the traditional approach in CS was:

1. Non-peer-reviewed "workshops" or "symposia". Place for position papers, work-in-progress, etc. Typically screened only for basic suitability for the event. In some cases authors may only submit an abstract of their topic, not a full paper.

2. Peer-reviewed but non-archival "conferences". Place to disseminate properly written up recent work in relatively short form [6-12 pages double-column], get feedback from other researchers, etc. Typically peer-reviewed by 2-4 reviewers, on the basis of a full paper (not abstracts like in some fields). The top 10-40% of submitted papers are accepted for inclusion (depending on the conference's selectivity), and then accepted authors typically have a month or so to revise their papers on the basis of reviewer comments, before submitting the final version for publication in the proceedings.

3. Peer-reviewed archival "journals". Place to wrap up a project or line of work with a publication that takes 3 to 5 conference publications, and integrates them into a full writeup of the project, tied into a nice bow for posterity suitable for archiving in The Literature. Peer-reviewed in the same way as other fields. Typically longer, 20+ pages.

But what seems to have happened is nobody really reads #3, since all the new work comes out in #2, the peer-review standards of #2 are already high enough for practical filtering, and #3 is years behind. And then people (especially Americans, especially in certain fields) just stopped writing #3 as often altogether. The view seems to be that tying several conference papers into a bow might be nice from the perspective of having a sorted-out archival literature, but at the present time, your peers have already moved on, and there are diminishing returns from tying those 3 already-published SIGGRAPH papers into a summary journal article, compared to working on new research for the next SIGGRAPH paper instead. It's even seen a little as "milking" your previous work: if you take your conference publications and turn them into a journal article, some people will assume you're doing it because you need the journal line on your CV for some kind of tenure/hiring committee, since otherwise why would you be spending all this time "upgrading" old conference papers into a journal, rather than writing new conference papers on new research?

Replying to self (too late to edit): One other reason it developed this way, I think, is because of the strong industry participation. Industry researchers often are willing to write a short conference paper: Usenix, SIGGRAPH, etc. But they usually (with exceptions) have little interest in writing up a 20-page journal article and going through a heavyweight revise-and-resubmit review process that might take a year. So the good industry research ends up mostly at conferences, and since some of the industry research is very high-profile (e.g. Pixar stuff at SIGGRAPH), it further cements the conferences' reputation as the place top research appears.

Yeap. Industry has to ship something useful.

That's because it takes a long time to publish in a journal (like 1.5 years or longer), and by that time the topic is old. Publishing in a conference allows you to bring fresh-from-the-oven results to your colleagues, along with the freshly-baked bread smell ;)

Journals are highly variable, at least in physics. I've experienced everything from acceptance after a week and publication within the month to pure hell that dragged on for over 2 years. The average is a few months and certainly not over a year! You'd better believe physics moves fast too. Preprints frequently gather dozens of citations before a journal editor even bestirs him or herself to send the submitted article out to referees! ArXiv is what everyone reads daily for the latest stuff, but always with caution. There's crazy crank stuff and crap in there too unfortunately.

Preprints are a great way to distribute results quickly without waiting for the gears of the peer-review process to complete their grinding or for the right conference to come up. However, when you get good referees the peer-review process can really put your work through the acid-test. We frequently get comments back that prompt us to do great work (along with many that just piss us off with busy work...). When it functions properly, the peer-review process bursts your institution's local bubble of thought-patterns and reminds you of how other experts in the field might think about the same problem. Occasionally it catches big mistakes or spots huge redundancies.

China is starting to publish a lot, but the standards aren't there yet. Some of their institutions do fantastic work while others really need peer-review to keep them honest. There are, of course, dishonest individuals pretty much everywhere who will take advantage of any flaws in the system. How do disciplines that eschew peer-review deal with this? When viewing preprints, I unfortunately place far too much importance on authors I know and trust. Sources matter when you can't really trust every article you read. Without peer-reviewed journals, how do you avoid potentially ignoring brilliant research by people unknown to you?

Reply on news.ycombinator.com