Hacker News Reader

Comment by Vaslo | Hacker News Reader

I agree with this article and have always thought about this but never voiced it.

I think a few things could improve the quality and discovery of published papers:

-After so many publications, could it be mandatory for random samples in that author’s publication be tested? I get that there is a limit on resources, but some advanced undergrads could do this with guidance.

-I would love to see some version of a journal of failures. That is, well intentioned research that had poor outcomes. It was so frequently in Chemistry research that my compounds were useless or the methods to synthesize them did not work, and it would be helpful to document that. Unfortunately, there is no “Journal of Failed Chemistry.” Only research that ostensibly make a contribution with a clear outcome get published. So much time wasted experimenting where you could save another scientist time and encourage them down another path.

View on HN

>It was so frequently in Chemistry research that my compounds were useless or the methods to synthesize them did not work, and it would be helpful to document that.

This mirrors my experience where a lot of the post-docs carried war stories of things like lab/country-specific humidity playing a role in synthetic methods succeeding (or failing). There were a lot of dark arts/tricks of the trade that people carried around with them: stuff like going that extra mile to dry things of water super thoroughly (even if it was not mentioned in the paper we were referencing).

Many "dark arts" aren't known as dark arts to the people who know them. They're just the things you always do, because if you don't do them, nothing works.

An approximate CS analog: Writing great commit messages, and using an SCM.

You don't have to do them, nobody writes it up because those who know regard it as trivial, but if you don't do them, almost nothing works. Everyone who knows what they're doing does it.

>An approximate CS analog: Writing great commit messages, and using an SCM.

This is definitely not the analogue. The analogue is always doing a clean install of the OS before running your experiment. Or only ever using Arch version x.y.z for replicating lab 1 and maybe a.b.c for replicating lab 2.

It's knowing all the magic undocumented JVM flags ahead of running the application.

Some you know to use as part of war scars/best practices. Some are just pure inside information from working in that lab or having a personal relationship with people in that lab.

There are a bunch of journals like that. If you google "Journal of Negative Results", you will see several for different fields.

However, for some I know, it's really difficult to get in, because there are many more journals of positive than negative results, and there are probably many more negative results than positive... so your negative result had better be especially interesting.

I recently noticed SURE journal, intent on publishing "unsurprising" economic articles.

I just wish there were more, or at least more journals rewarding sound methodology instead of unsound "results". Who knows how much effort is wasted reproducing sound, but unremarkable studies simply because they weren't published

Partly the issue is because of the naming... Like "poor outcomes" and "failed"...

Obviously there would be no awards for the poor outcomes and failed results and we would be back to square one where no one is incentivized to look at them positively.

What are you waiting for creating one? Motivate a few scientists to form a credible review comity, fill one paper [1] and voila you have the journal you dreamt about on a recognized and perennial web platform.

This is of course some work, but if you really want it to exists it’s better not to wait for someone else to do it. Also allowing anonymous authors would help I think (to avoid the name being associated with failure).

[1] https://www.openedition.org/15974?file=1

Edit: openedition.org is for humanities, but there probably is alternative for other fields.

> What are you waiting for creating one?

Creating a journal is a tremendous task, and only highly respected researchers have any chance.

> if you really want it to exists it’s better not to wait for someone else to do it

Again, most people simply cannot create a credible journal. Let's be realistic here.

On failures: many years ago I stumbled across the idea of a `CV of Failures` where you list all of the accomplishments you missed/were rejected from, etc. I think it's a very humbling idea and can see some applicability here.

Example: https://www.princeton.edu/~joha/Johannes_Haushofer_CV_of_Fai...

This is a variant of "the carrot and stick" approach that happens to be very short on the carrot and be very large on stick.

Very unlikely to be effective.

Time and again people think that when they have a problem it must mean the punishment was not severe enough. It won't work.

The problem is that good science is not rewarded enough. Nobody rewards you for having published reliable stuff five years ago.

Unless the GP edited, there's no suggestion of punishment in his comment.

maybe a more serious version of the journal of irreproducible results? https://en.wikipedia.org/wiki/Journal_of_Irreproducible_Resu...

Reply on news.ycombinator.com