How scientists fool themselves and how they can stop
nature.com
nature.com
Either you confirm previous results, which is nice but not something you can publish or write a thesis about; or you find they are wrong and then you have to tell the authors, which is something they may appreciate or not, and be exhaustive to prove it, which is something in which you may be interested or not.
What I would like to see is that PhD students were required to reproduce some previous results as part of their thesis. This would increase the quality of scientific results and would prepare the candidates to do their own research (at difference of now, that they are put on front of a laboratory to get "interesting results" and they don't know what to do with it).
Take this recent publication [1]:
> For water oxidation, the photoanode surface was protected from corrosion by a 62.5 nm layer of amorphous, hole-conducting TiO2 that was grown by atomic-layer deposition (ALD).
> The TiO2 layer significantly improves the stability of III-V photoanodes in a tandem structure for water oxidation while the tandem structure produces sufficient photovoltage to sustain the efficient, unassisted production of hydrogen by water splitting in aqueous alkaline electrolytes.
They discovered that coating the photoanode with 62.5nm of TiO2 helps stabilize the reaction. But who knows how many materials they went through to get that one? And how many different coating thicknesses they tried before settling of 62.5nm? This tech could be next-gen solar, would be great to see the global rate of discovery increase. Perhaps YC Research can start this trend?
[1] Joint Center for Artitificial Photosynthesis - http://authors.library.caltech.edu/59897/1/c5ee01786f.pdf
I think a couple of bigger issues are (i) we reward the first person/group to reach a conclusion, so there's incentive to be sloppy and get your name on the research first and sort out the (hopefully minor) problems later and (ii) there's an expectation from the general public that scientists have to be "productive" all the time. Productive is of course defined as producing new and exciting results, rather than carefully verifying old results, or trying to fill the gaps in previous work. See how much pressure there is on scientists to do "transformative" rather than "incremental" research.
I don't think we as a society have a solution to these problems even outside of science. To pick a familiar example for HN, programmers who write overly-complicated messy code are often rewarded over those who write simpler, cleaner and well-architected code because the former looks like more work. Similarly, there's little interest in writing test benches or documentation in open source projects, because there's little reward or incentive for these things. OpenSSL pre-heartbleed era is a perfect example of all the wrong incentive systems working together to produce a steaming pile of bullshit that everyone involved in the field kept claiming as being the most secure solution out there.
Of course, other colleagues are not so lucky, and getting funding is an important part of their day-to-day jobs, specially once you reach higher positions. But it is a necessity, not an instinct.
(I heard a lecture once from a researcher who was on Wall Street, made millions, then switched to femtosecond lasers. He worked out of a university, and funded his own lab. So such people do exist.)
If there are differences, are they measurable?
Conversely many of those funding their own research are doing so because they cannot get any other funding, which may be an indicator that they aren't very good.
Really? I seriously question that "because". If I had FU money I would definitely start my own lab and pay for research out of pocket. Science is fantastically fun if you don't have to care about funding (and by extension tenure) and only publish because it's nice to get feedback from your peers (rather than because you need X papers to get your next grant and keep your job).
Conversely, writing grants is a total bore.
A "because" I would be more willing to buy: if you made that kind of money for yourself, odds are you spent your prime years achieving that and therefore haven't dedicated the time to science that's necessary to be at the top of your field.
... at selling. In my experience the skill sets needed for aquiring funds and those for doing research don't have much in common.
In psychology on the other hand (a field I have a huge amount of respect for, incidentally), we're trying to ask questions about human behavior; everything we ask about is taken within a cultural context, predicated on assumed levels of "normal" behavior, and even assumes that humans /are/ repeatable in some sense (and a lot of psychology studies may not be repeatable at all...)
That repeatability question is a big one, I think: we're a lot more sure about our ability to prepare equivalent circumstances with silicon-28 atoms for an experiment than with people.
There's an absolutely unambiguous distinction between the "hard" and "soft" sciences
Social scientists hold themselves to fundamentally different standards on things like repeatability and predictive power and methodolgy than other fields. And whenever that difference can be quantified (e.g. statistical results), "different" means "lower". It's just a fact, and it's one most sociologists will freely admit, for example. You simply can't get the same sorts of results on humans that you can get on e.g. chemical reactions.
Unfortunately, soft/hard comes along with a value judgement that's not necessarily appropriate. But pretending like there isn't a clear difference between what are now referred to as the soft and hard sciences is disinegenious.
For instance, economics has, I would claim, far weaker levels of model validation than engineering fields, but often is presented as a hard or 'close-to-hard' science.