Comment by alextheparrot | Hacker News Reader

Comment by alextheparrot | Hacker News Reader

I’m glad this was posted. I’m writing my personal statement for applying to PhD programs right now, after spending a few years in big tech and both the concerns as well as the benefits resonate with how I’ve modeled my expectations.

I think I’ll end up in a similar place - hyper-focusing as a graduate student and then transitioning out of academia after. One understated part of academia is that for a lot of fields (Such as the bio-adjacent fields I’m interested in), academia is one of the few places willing to take on trainee risk. I hope we can keep academia as a place where we’re willing to take risks to educate, which I think starts with reforming the processes informed by the critiques this post levies.

One thing I would advocate for is forcing pre-registration of experiments. Publishing a design for experiments with an intent, a hypothesis, and a afterwards a quick summary of the results. That process for FDA trials has naturally led to a large buildup of both positive and negative results, while also inhibiting post-hoc statistical baboonery.

I’m for pre registration for some fields with big large experiments.For other fields (robotics is what I have in mind),I think the experiments can be so rapid that pre registration would be a real hassle and slow down progress.

This seems to be a separable point, though. The issue here is not the act of pre-registration, but the friction it imposes. I know most of the experiments and research I have done could have been summarized at the time in 10 minutes - to the effect of "I will manipulate variable X over time ranges Y, this is supported by <source or professional explanation> and I expect Z", or even "We performed a gradient over variables X and Y expecting to modulate Z and looked for phenotypes not previously described for further characterization".

My experience in research is that there is very little pre-commitment on this level of granularity. Grant applications should be at the level of "We hope to study X because it is known to be correlated with Y and are hoping to elucidate a mechanistic relationship between these two". The experiments which go into proving such should more closely resemble mathematical proofs, is a viewpoint I currently hold.

why can’t you gpg-sign a text file and then get a bitcoin timestamp receipt? There is no need for bureaucracy.

Pre-registration is what funding committees and proposals are for. You get the funding, the experiment is approved, now go do it and publish the results.

In my experience [0], the grant funding tends to be something more akin to "We will attempt to further characterize area X, using tools Y and Z, which we expect to be involved in important area A, but has not been proven so, yet". The granularity I'm speaking to is at the lab-notebook level, give or take. If I'm performing an assay, formally documenting the premise, evidence supporting that premise, and result under that grant heading would mean we could easily publish both negative and positive results for a given study.

One of the areas I helped write a paper for as an undergraduate was later proven to have a completely different mechanistic understanding than we proposed. I think having this more detailed set of observations would have helped both us and any people citing our paper come to this conclusion earlier. Physics is great at this [1] - seeing "dark matter" or something unexplainable is an opportunity, but many fields treat presenting something as 'unexplainable' as a weakness.

[0] I don't have experience in particle physics or anything of that nature where you're working on constructing something like the LHC

[1] Once again, outside observer bias

Reply on news.ycombinator.com