Hacker News Reader

Comment by danielweber | Hacker News Reader

I was very surprised to not see Jason Mitchell at Harvard Univeristy's attack on the concept of replication: http://wjh.harvard.edu/~jmitchel/writing/failed_science.htm

The thing is amazing to read, in the "someone who thinks he's a scientist actually wrote this?" sense, like this gem:

Because experiments can be undermined by a vast number of practical mistakes, the likeliest explanation for any failed replication will always be that the replicator bungled something along the way.

View on HN

Wow, this is quite stunning to read. He actually baldly states that the only reason for experimenting is to confirm what you already believe, and that you should therefore just do as many experiments as it takes to 'prove' yourself correct, hide evidence of the failures and then stop - forever.

This was linked from the article, BTW - and the responses they also linked to are dead on:

http://mchankins.wordpress.com/2014/07/12/do-not-replicate

http://vserver1.cscs.lsa.umich.edu/~crshalizi/weblog/1102.ht...

That statement reveals a stunningly large unconscious bias. Let us assume that Study A produces result X, and Study B failed to reproduce it. Let us take for the sake of argument an assumption that one of these studies is flawed.

Which one is it?

You can not tell.

There is an old saying: "A man with a watch knows what time it is. A man with two watches is never sure." - http://en.wikipedia.org/wiki/Segal%27s_law

I remember reading that and reeling, shocked that anyone could write and sign their name to that. Whenever I am pessimistic about my chances in science, I remember that that guy has a job at Harvard.

He is a scientist; unless you're also a domain expert in his field with at least as much expertise as him, or a newcomer with radically impressive work, you can't just not call him a scientist because he disagrees with you. If a domain expert disagrees with you, it is probably because you are wrong. That's just the base rate and it's probably true here as well.

Traditionally, replication hasn't been done in psychology because the experiments have to be set up in really clever complicated ways in order to tease out effects. Even the most dedicated replicator can't ever fully replicate a psychology experiment. You'd have to ship the actors used as confederates across the country to do that and of course nobody does that. It's foolish to think any psych study has ever been replicated.

The way psychologists deal with this is the concept of converging operations. This has to do more with being clever and considering implications of a thing you're trying to study with respect to other theories you have much more evidence for and in different situations. If you think stereotype threat is a thing, for example, it would make sense for stereotype boost to exist if the stereotype is positive. If a priming effect speeds up the recognition of "doctor" after a subject has been shown "nurse", maybe recognition for "warmth" will be slower after a subject has been shown "frozen". If people only encode based on schema variables, they'll be unable to remember information unrelated to a schema even if they're provided with congruent encoding cues.

Converging operations is just studying the same concept from different angles and seeing if you keep seeing consistent results. If the results aren't consistent the hypothesis you're investigating will, in fact, die. Psychology is an advancing science as of right now. If people were just publishing random noise there'd be no reason for concepts to fall out of favor.

I think people on hacker news tend to wildly underestimate the degree of skill involved in experimental psychology. So much of statistics, even, has come out of psychology because of the need to refine and clarify what effect you're seeing.

Alternatively, if you think because of this existing academic psychology is lacking, you're in a fantastic place to be. Do literally anything to get some seed money, just working in a standard SDE job would be fine, and then run some small experiments (and replicate them, of course) on learning or addiction. Then write the next Farmville or build the next Facebook. If there's really that much low-hanging fruit, go out and pluck it. Put up or shut up.

Now wait a second. When a psychology study comes out, it claims "This experiment shows stereotype thread reduces performance of Honduran men on a Math test." Such studies rarely claim "This experiment shows stereotype threat reduces the performance of Honduran men on a Math test when Jill the skinny experimenter repeats the code words."

If using different experimenters yields a different result, it means the effect being described is probably not as robust as the experimenters claim. The cause might be Jill's shifty eyes rather than stereotype threat.

That's a replication failure, in the sense that it shows the claimed effect is far weaker (or causally different) than the original study claimed.

I would say that it's typically assumed that there are confounding factors in any experiment. An experimenter has to try to minimize that, but you're still not going to be able to replicate a full experiment.

If you have more than one researcher administering a test in a stereotype threat experiment, that'll reduce the likelihood of that being the confound. If other studies (not replications, but their own experiments, with their own, different approaches to study the problem) agree with the first experiment, the effect is likely real.

I would also say that you're viewing the entire system far too antagonistically. Nobody goes into academic psychology to get rich off of it.

But if you're not minimizing the confounding factors enough that the result can be repeated, your experiment failed. Your hypothesis has neither been provably confirmed nor denied. At that point you are, at best, still gathering evidence, and publishing results would be an error.

>Then write the next Farmville or build the next Facebook. If there's really that much low-hanging fruit, go out and pluck it. Put up or shut up. //

Your reasoning here is as fallacious as the idea that unrepeatable experiments are good science.

Your psychological hypotheses on addictive mechanisms can be entirely false and yet your game can still be addictive. Maybe the Skinner Box-like environment you create has a particular applicable demographic outside which it is largely impotent; but failure to repeat the experiment isn't seen as contradictory to the hypothesis and so you fail to address different markets with the specific sort of addictive elements that suit them best. You'd be leaving a heap of money on the table [for shame /s].

The ancient Greek idea that gravity worked by the attraction of Earth-kind substances to the Earth itself was falsified and yet things still fell down.

Would you say that a single superstitious action (finger crossing, say) coinciding with the desired outcome proves that finger crossing works? Do you think that those who ascribe to such a progression of logic as demonstrating sound scientific thinking, are being scientists?

IMO observations and experiments that can't be repeated are of the utmost importance in furthering knowledge. Truthfully reported anecdotes are valid data. But unrepeatable events have limited scientific value in and of themselves, they can not show statistical significance nor demonstrate that a hypothesis should be held to be true as an objectively valid scientific theory.

You're conflating a number of ideas, but the overall trend in your post is towards a debunked theory of science called positivism. You should research that.

Is your contention that [social] psychology need not rely on an empirical approach. How then are you to claim it's "scientific". Perhaps the best question to ask here is what do you consider, succinctly, to be the basis of the scientific method?

I did indicate that I'm happy to accept other methodologies as beneficial and am certainly not wedded to positivistic approaches in all fields of human understanding. Are you perhaps pushing the meaning of "repeatable" in the current context beyond the bounds of "supporting the same breadth of conclusion with equal confidence".

It is however not scientific to rely on unrepeatable experimental results. If an experiment is unrepeatable then the conclusions are proved false. Note that the question at hand is not whether the initial experiment is valid or useful. Science after all is axiomatic and the world, generally, non-deterministic so far as one can tell [though I know there are many dissenters on this point].

Well "pbhjpbhj," like I said, you're confusing logical positivism with science. Positivism, the idea that we can only believe propositions which are proved correct, lost favor in the late 1800's. Modern science, including social psychology, is in the vein of scientific theory called operationism. Like I said before, you should research this, because you don't know what you're talking about.

We're not talking about multiple metrics, the exact same metric is being used in empirically identical circumstances and the results are conflicting. The only way to square that position is to assume the conclusion is wrong or the data is wrong. Essentially the epistemological underpin isn't important; operationalising a system no more allows for unrepeatability in empirical data than does any other consistent scientific method - after-all if unrepeatability were allowed then the logical basis would be self contradictory.

Rather than arguing from abstract implications about my assumed knowledge why don't you address the question at hand?

Don't conflate Experimental Psychology as a whole with the subfield of Social Psychology, or the particular social priming community. While it is true that the "skills" argument and "conceptual" replications are well-established methodological pillars within this subfield, they are far from standard outside of it.

This is why this string of multiple replication failures and fraud scandals have turned the spotlight to these practices from within the discipline, with reasonable and high-profile people openly questioning their validity and pushing for a whole "replication movement" [1-4]. So it's not just HN commenters, you know.

[1] http://www.nature.com/news/replication-studies-bad-copy-1.10...

[2] http://chronicle.com/article/Power-of-Suggestion/136907/

[3] http://www.psycontent.com/content/311q281518161139/fulltext....

[4] http://www.nature.com/polopoly_fs/7.6716.1349271308!/suppinf...

I wouldn't say that converging operations is limited to social psychology. Virtually all memory research (cognitive, not neurological) is conducted in the same way. As various other areas get more neurological basis I think psychology will move away from it in favor of more precision measurement, but it's still valuable.

Absolutely. But outside of Social Psychology there are strong, genuine disagreements --to say the least-- on whether "conceptual" replications are a valid source of converging evidence at all. See for instance [1,2] for typical reactions.

[1] http://neurochambers.blogspot.com/2012/03/you-cant-replicate...

[2] http://psycnet.apa.org/journals/gpr/13/2/90/

Reply on news.ycombinator.com