I get really sceptical about these social sciences studies, especially after the recent round of replication attempts that failed.
Human beings are complicated and difficult to study, so I can appreciate the challenge that social science researchers have in front of them. But, propagating conclusions from a field of research that has a 33% reproducibility rate is just negligent journalism. It's not true just because it's in one study and it matches our intuition or something we read in a Malcolm Gladwell book.
1. Over half of psychology studies fail reproducibility test. http://www.nature.com/news/over-half-of-psychology-studies-f...
Well that's a positive way to put it. I wonder whether it is possible to find steady cause and effect patterns at all.
Same goes for economy (which IS basically a proxy for human behavior). You can certainly have a go at documenting its history but the present and future are probably not predictable because they are subject to evolutionary responses. When I had to take an economics 101 class negative interest rates were out of the question and look what we have now.
And you're aware that the core part of that documentary is based on an (AFAIK) unreproduced study on psychopathy - by somebody who likely has a vested interest in psychopathy being a problem. (Robert Hare - see www.hare.org)
I'm all for questioning the accuracy of studies, but I'd hope we'd use somewhat higher standards than that.
So the criticism of the criticism seems to be quite overstated. If the study can only be replicated at a single university (or even all in a general area), it means that we need to be very careful in generalizing the results. So to the extent that people take a study of one very specific group and generalize to an entire culture (or worse, to all humans), the criticism sticks.
Back when I was getting my psychology degree, there was a joke that 90% of the findings in psychology can only be generalized to broke American college students who are willing to be lab rats for beer money. While that overstates the problem, there is still a lot of truth in it.
The response to the paper points out few examples where the replication was not judicious in picking a sample that minimized its influence on the measured effect. Even besides that, the original study had sampling and scoring procedures which contradicted its original aims.
Lastly, sensationalist headlines of "more than half of all papers are not replicable" are blatantly wrong and not what the original paper even states.
As to your point, I agree that only being able to replicate in one sample type is grounds for skepticism, but more replications have to be done to ensure the replication sample is not the one that is flawed. One can't really draw conclusions unless the replication is repeated a large number of times across a wider sample space.
Link to response paper in case anyone's interested: http://projects.iq.harvard.edu/files/psychology-replications...
All this shows is that psychology is less useful that people thing because any data is likely only applicable to the given group. Even if we say the studies are still good, this means that psychology loses the ability to be generalized to others groups beyond the one studied. And given how often psychology has that done to it, we must question every finding all the more.