Hacker News Reader

Comment by nonbel | Hacker News Reader

> "Add reprodicibility criteria for peer review."

Passing peer review is something used as a substitute for reproducibility. Just like testing a null hypothesis is a substitute for testing the research hypothesis, and citation count is used as a substitute for making precise and accurate predictions.

The procedure (scientific method) that has lead to all the great stuff we have around us today, I'd even call it a pillar of civilization, has been undergoing a piecemeal replacement since approximately WWII.

View on HN

> Passing peer review is something used as a substitute for reproducibility

Wow, pretty low bar for science. You can have peer review for the actual results and what they mean, as well as them being reproducible.

> The procedure (scientific method) that has lead to all the great stuff

I don't think you know what the scientific method is.

Scientific method (basically me just paraphrasing Imre Lakatos):

1) Explore and describe aspects of the world in detail, figure out what situations naturally arise or that can be devised that produce consistent, stable phenomenon.

2) Abduce (guess) an explanation for these consistent, stable phenomenon.

3) Explore the logical consequences of assuming your guess is correct. Figure out a few otherwise surprising (ie, inconsistent with other explanations people may have) predictions that can be deduced from it.

4) Collect data and compare it to the predictions generated in #3

5) Discard the guess or modify it to make it more consistent with the data produced in #4

6a) If the guess is modified, Go to 3.

6b) If the guess is discarded, Go to 2.

A lot of cancer research seems to be failing at step 1, ie they can't even get into the "testing otherwise surprising predictions" loop since there is no consistent, stable phenomenon to trust.

From wikipedia:

1. Define a question

2. Gather information and resources (observe)

3. Form an explanatory hypothesis

4. Test the hypothesis by performing an experiment and collecting data in a reproducible manner

5. Analyze the data

6. Interpret the data and draw conclusions that serve as a starting point for new hypothesis

7. Publish results

8. Retest (frequently done by other scientists)

Emphasis on 4) and the "reproducible manner".

Otherwise what's the point really? We would resort to trust or belief in one's sayings.

The wikipedia one is ok, but really most of that is stuffed into my #1 and it treats the "forming and testing a hypothesis" part too superficially. What did I write that makes you think the two are in conflict?

I wrote:

"figure out what situations naturally arise or that can be devised that produce consistent, stable phenomenon."

This is the same as figuring out the "reproducible manner".

I don't think you understood what I took issue with in your post.

You said:

> Passing peer review is something used as a substitute for reproducibility

And what I'm saying is "No, it's not". Reproducibility of the results is the most important aspect of the scientific method, otherwise, as I said above: "what's the point really? We would resort to trust or belief in one's sayings."

Oh. I never said that was a good thing. Institutionalized peer review as its done today is a relatively new thing, introduced post-WWII. Ie, that statement was descriptive, not normative.

https://www.timeshighereducation.com/features/peer-review-no...

OK, thanks for clarifying, I obviously misread your comment.

I think you mean "proxy variable", not "substitute". That is, it's a variable which is correlated with the one you really want (quality, truth) but can't actually measure due to difficulty.

Reply on news.ycombinator.com