I'm not saying research fraud doesn't matter or isn't worth the stories written about it. I'm saying that people commenting on it generally don't have any sense of its scale, and fill in some very weird blanks about it.
I'm not saying research fraud doesn't matter or isn't worth the stories written about it. I'm saying that people commenting on it generally don't have any sense of its scale, and fill in some very weird blanks about it.
It is very much a crisis and one that academia has no interest to solve.
In a sane world we'd start with figuring out what the mystery meat in the version of pytorch used for training is and work up from there. There is zero funding for building up the required infrastructure for doing actual science instead of whatever the fuck you call the current mess.
Basically anything on a more complex dataset than cifar is probably a statistical fluke because of hyper parameter tuning. To find this out you will need to spend three months building up all the bit rotten code that doesn't work from the paper. To draw the conclusion that everyone is full of shit you'll need to waste years of your life.
Instead, we're making astounding progress at a rate that no one really would have expected.
Learning to replicate papers is a skill. It's just a matter of practice. You'll get it if you try.
A lot of small to medium sized companies fighting for mind share also release one script deploy pipelines for their work, which are better than the academic releases.
It's western universities and big tech that releases at best bit rotten code and at worse no code what so ever.
It's hard to even name mediocre advances that come from anywhere else. As well as the entire hardware and software stack that we have.
You clearly have some sort of political agenda here and not a technical argument.
Transformers weren't scaled at google because it would eat their ad business. We could have had current level AI in 2020 if they'd been serious about it. They weren't and everyone who worked on it left when the gold rush started without having done anything meaningful in google.
Computer vision, which is the field I've been working on the past few months, is even worse.
Perverts in their basements are getting better than sota image recognition so their wifus have realistic vaginas, and writing tutorials which provide more working insights than any scientific paper.
This isn't a new thing. I dropped out of my PhD working on something similar to AlexNet in 2009 because everyone thought I should just use gradient boosted decision trees like everyone else at the time. I couldn't get $10,000 grant to buy a machine to run the training on.
It isn't just me. Yann LeCun's networks were running the postal service, but he couldn't get a paper published anywhere because it wasn't a hot area to publish in.
But, yeah. Tell me more about how great Western academia is.
There was decades of “but everyone is doing it!” in the system, so much so that fraudsters became presidents of Ivy League institutions. The reckoning has not come.
And that's not to speak of things like biotech which is full of photoshopped images of fluorescence and whatnot.
People are counting on readers not to know or take the time to learn how much research is happening, so that individual instances of the problem (in the United States) seem systemic.
* The post is a sample of papers across many fields
* It describes a method to determine whether widespread p-hacking is occurring in a field
* Widespread p-hacking in a field will lead to unreproducible results (because the results are an artifact of the data rather than reflecting any underlying cause)
* You said: The replication crisis stuff is mostly a message board fixation. The problem we have is that people unfamiliar with the field don't have a sense of what the denominator is, only the numerator.
* The approach in the post uses a statistical method across many papers across many fields. It does not rely on the existence of some number of non-replicable results (in fact, one should expect some percentage of results to be non-replicable).
* Therefore, it doesn't rely on "the numerator" so to speak, but a sample of all papers in a field to characterize the field's replicability.
You're a smart guy. I follow your posts here and on Twitter. So I can only assume that either I've flipped some bozo bit or something in my communication style is abstruse. I don't understand why you're saying that p-hacking has nothing to do with the replication crisis (most would expect the two to be quite related), or that saying "the replication crisis is real" means I am claiming that individual papers exist that are non-replicable (which is fine, but not a replication crisis). In either case, we're clearly getting nothing out of talking to each other so I might as well leave off here.
Either way: people are responding to specific stories about fraudulent research, and my point is that the numerator of those stories is infinitesimal compared to the denominator of how many papers are published in these fields annually, by how many separately-funded R1 research teams.
I'm really not interested in being boxed into a debate about whether p-hacking matters, or scientific fraud exists. Fraud is bad! I'll just keep reiterating that the optimal level of fraud in a large system is almost never zero.
† That article is about a bunch of different fields, and I'm commenting only on a very specific subset of them; fuck do I know how well social science is operating right now
I've been trying to think about what you meant since you wrote this earlier in another comment, and I'm having some trouble wrapping my head around it so I think I'll just ask (if you'd be so kind to explain).
What makes some level of fraud more optimal than zero fraud?
Grinding all scientific research to a halt to "true up" the statistical reporting used for otherwise valid and important research results would be a very good example of an intervention that would (a) "reduce fraud" and (b) leave us all significantly worse off.
Whether it's mine processing, agriculture, bank lending, { insert domain } there are always a number of drags on profit ( fraud, equipment wear and tear, petty theft, holes in fences ) and generally always a threshold past which the cost of pursuit of perfection exceeds the decreasing marginal benefit.
That is to say, Zero is the optimal in an imaginary system, where cows are spherical and you can make all the assumptions you want.
Once you start working with real world systems with more and complex constraints, the simple story goes away, and perfection is not possible.
The phrasing challenges the reader to question the model with which they are using to define optimal.
"Conforming or limited to the simplest, nonfigurative, or most obvious meaning of a word or words. "
"Word for word; verbatim."
Taking the sentence in question verbatim would be "some fraud is optimal". The nuanced (non-literal) interpretation is what Thomas wrote.