Comment by jboggan | Hacker News Reader

Comment by jboggan | Hacker News Reader

Parent Full thread

Spot on with the alchemy remark, I've made similar comparisons before. Coming into bioinformatics/computational biology with a strong discrete math background I found a lot of professors excited to work with me until I started telling them how their ideas and models and experiments didn't imply what they wanted them to. Just like the startup world is awash with "it's like Uber for X" the biology world is full of "let's apply X mathematical technique to $MY_NICHE" and somehow this is supposed to always generate novel positive results worthy of publication. Then you tell them that you applied such-and-such mathematical/statistical model to their pet system and that the results contradict their last 10 years of published papers . . . and they ask you to do it again.

I remember one professor studied metabolic reaction networks modeled with differential equations. The networks themselves were oversimplifications and relied on ~5N parameters (N being the number of compounds in the network). The problem was that while all the examples in publication converged on a nice steady state (yay homeostasis is a consequence of the model!) it was trivial to randomize the parameters within their bounds of experimental measurement and create chaotic systems. Did this mean the model wasn't so great? No, it just meant those couldn't be the real-life configuration of those parameters . . . sigh. And now I'm a data engineer and no one asks me to get data from an API that doesn't actually provide it and I'm much happier.

"it was trivial to randomize the parameters within their bounds of experimental measurement and create chaotic systems. Did this mean the model wasn't so great? No, it just meant those couldn't be the real-life configuration of those parameters . . . sigh."

Why is this sigh-worthy? Maybe the model is still ok. Maybe some of the parameters are constrained in nature in ways that you/we just don't know about yet. Of course, it could be that the model is bad, but you act as if your results, as stated, demonstrate this. Which they do not, as far as I can see.

I hoped that it wasn't as bad in computational biology, or ecology, or any other biology field where systems and models are actually defined. It saddens me to read that your experience was as bad as mine...

I'm a CompBio PhD student, and my experience is that folks in that field are much more careful with statistics than in, say, molecular biology labs, but it varies from lab to lab. My PI is exceedingly meticulous about stats -- for instance, we don't report p-values, but rather entire distributions -- but that's because our work is all in silico, so it's easy to run tons of replicate simulations. Wet lab work that's finicky should definitely be held to high statistical standards, but I don't think it's fair to presume everyone in the field guilty until proven innocent.

But wet lab scientists should be even more careful! They have way less control over the system they're trying to study than you do, so stats are the only security net we have to even attempt to do anything with the data we produce.

I also agree on the innocent until proven guilty part, but by now I've seen and talked to hundreds of people with the best intentions, who do not realise how important careful examination of the data is, so I'm growing a bit disillusioned.

I think it is entirely fair to presume biologists incompetent and sloppy until they have proven otherwise.

(My impression from admittedly limited contact with biology students and from browsing through the occasional paper is that most of them barely approach mediocrity from below.)

Reply on news.ycombinator.com