Hacker News Reader

Comment by harry8 | Hacker News Reader

Fails to touch on the perverse incentives in academia, "publish or perish" etc. Torturing a dataset to find a p value that a journal will like (or equivalent stat measure) is better for your career than not publishing a paper that will be discredited in time. You have no incentive at all to decide "my results are unconvincing at this point, I'm not going to submit them" and every reason to write them up as a useful contribution to human understanding even if you kind of know, deep down, it really isn't. Especially if you're not senior...

View on HN

Some days ago there was a post here on HN about a post doc who failed to get tenure.

And every other comment was like: “What did he expect, he had much less than the usual two papers a year.”

So it seems that even here on HN, the mindset of quantity over quality still persists.

Those were not normative statements.

That's possible of course.

The problem really is: there is no alternative (which I know of). Having a number of papers in well known journals is everything we have to gauge the quality of people that look for a life time position. It’s sad but true

No, the number of papers is an okay metric, thr problem is that journals don't like to publish honest negative results

There are a few journals that will promise to publish your results based entirely on your pre-registered plan -- e.g. whether or not you find the correlation you were looking for.

In the long term, it seems like journals that publish lots of falsified papers should be punished, and journals that don't (e.g. because of a judge-upon-pre-registration policy) should crowd them out.

Pre-registration is great, and I think it even helps researchers/scientists/post-docs to submit something (it doesn't have to be perfect, after all, it's just an experiment design, and they don't have to worry about massaging the data to have promising results to report) and then stick to the topic, and then carry out the experiment as well as they can, and gather as much high quality & fidelity data as they can, and do the analysis according to the plan, and then report it. No extra pressure to think about how to frame what you "found".

Though this will inevitably lead to the problem that grant boards face, that it'll be a lot harder to differentiate between proposed experiments. And it'll be even harder to do boring stuff. (So if we assume all submitted plans are sound, they have to publish them all. Though then we'll have journals based on how strict they are with experiment design requirements, 1 sigma, 2 sigma, 5 sigma, etc.)

that author’s problem was that they had been a postdoc for 23 years. That duration alone raises fatal red flags.

It's possible you're right, but that's not what people were saying IIRC.

That's exactly what they were saying. Go read the thread again. They were being prescriptive of academic careers, not descriptive of the author's inherent merit as a researcher.

and to recurse a layer, i’m being descriptive of the prescriptivists that i know to exist and i easily believe did that guy in.

It seemed obvious to me that those comments weren’t extolling any merit of the “two papers per year” heuristic, but instead were just cynically resigned to the fact that that’s just the reality of it (good or bad) and so someone pursuing tenure has to be planning on dealing with being evaluated that way (even if it’s a bad practice generally).

No...that's not what it seems. I remember reading all the comments in that HN thread on Friday. The top comments were dispassionately explaining the reality of academic career trajectories. That's a basic reporting of facts, not a normative claim that quantity is better than quality. They did not themselves say you need to publish more often to be a good professor; rather they discussed the incentives that lead this to be the practical reality of the field.

We need to find resources to fund "Failures in Science" journals that exclusively seek to publish interesting research that went nowhere. Personally I'd find these far more interesting to study. "Here's some background. Here's a pretty logical, plausible hypothesis we came up with and how, here's our experiment, here's our results, here's our thoughts as to why we were wildly wrong."

Instead of allows failures (which would be also good btw), i would increase the time at disposal needed to pubblish something worthwhile.

The greatest thinkers in history had no pressure outside their own urgence to solve/explore. Surely, not everyone is Galileo but today science need that kind of freedom.

There is a new conference in cryptology that promotes exactly this: CFAIL.

From their 2019 website[0]: "Do you have insightful and exciting work sitting in a drawer somewhere because it never quite panned out? Are you willing to share your failed approaches so that others can learn from them without having to re-travel the same road? Are you tired of reading papers that pretend the incremental result they happened to achieve was well-motivated and was their goal all along?

We are! That’s why we are founding a new conference: a place for papers that describe instructive failures or not-yet-successes, as they may prefer to be called."

[0] https://www.cfail2019.com/

We have that. People don't use it.

PLOS e.g. specifically stated multiple times that they'll publish what meets their quality standards regardless of a positive or negative outcome.

The problem is: Even if you publish failed research it won't get cited as much. And people still use citation metrics to evaluate "quality" of science.

Just having journals that publish your "failed" research is a good start, but it's not changing the incentive structure.

Agreed. This isn't a one-action problem to solve. But we need to lay the foundation for alternative incentives to be possible.

The FADS workshop[0] at VLDB was a nice attempt at this although I think the work ended up focusing on known historical failures as opposed to failed work that was not otherwise published.

[0] https://fads.ws/

The problem with a failed experiment, at least in machine learning, is that it's not always clear what caused the failure. Was it not enough training, was there some small "trick" the model could have used, were the hyper parameters off etc.

There are an infinite set of configurations that could fail, and it's not sufficiently useful to know they failed without understanding why they failed. And analysing failures in a useful way is an extremely difficult and fundamental problem. On the other hand, a successful experiment is an extremely rare event and hence interesting by itself.

This is the unfortunate truth. Furthermore, those who take more time to find a general, robust and theoretically sound result (beyond just the p-optimized publishable result), get filtered out of te tenure positions as not have a high impact factor, despite their papers in fact having a higher impact.

You may have meant "bullshitable result".

They aren’t mutually exclusive

This is the real issue.

Getting data is always expensive in any field so it's much easier to analyse data that already exists as you don't need a large grant application.

Furthermore, I think the use of certain ML techniques may be akin to resume-driven development particularly for PhD students given that the career prospects in Data Science in industry (and using the HR buzzwords like AI, ML, Deep Learning etc.) are much better than the thin pickings that remain in academia in many fields.

I don't blame the students too much. If you get into a PhD program, it's very hard to change fields or even specialties without starting over. A simpler tactic is to weave a field of interest into your existing research project, then market yourself in that field when you graduate.

I did the same thing as a physics student, but with electronics and programming, which furnished me with a marketable resume.

Publish or perish is gone. The NIH is no longer the funding source of choice, now it is pharma or private donors. These people want large impressive datasets, not publications. You can rent out data to pharma without having to publish anything. These don't even have to be useful datasets, mere size is enough to make them valuable to greedy prospectors with lots of cash. Then they can plumb them for bunk results ad infinitum.

Seems like we need general solutions to overfitting, not just in machine learning.

Didn’t we used to value negative outcomes?

Reply on news.ycombinator.com