Hacker News Reader

Comment by hluska | Hacker News Reader

I've always had problems with fields like psychology. Part of the field revolves around hard science - neuroscience researchers (as one example) use tools like fMRIs, etc to capture observable phenomenon. On the other hand, there is the art of psychology - social/developmental psychologists use dubious tools like surveys and interviews to try and prove a thesis.

I think that the average Hacker News reader could write a survey that would 'prove' that gravity doesn't exist. What kind of experimental integrity do surveys have?

And then there are participant pools. These pools are primarily composed of undergraduates (who often earn bonus marks/money for participating).

Add in a strong publish or perish mentality and you can see some serious problems. A whole lot of researchers are using dubious methods on a set of participants that do not adequately reflect humanity as a whole.

View on HN

There are some issues I (as a psychologist) would have with your assumptions.

Firstly, fMRI and neuroscience research is probably the single biggest source of errors and shoddy research in psychology. Its a combination of really small sample sizes, poor statistical tests (brain regions are independent of one another, really?) and huge amounts of data dredging to find significance. See for a roundup: http://escholarship.org/uc/item/51d4r5tn

Interestingly enough, most of the observable pheonomena you note in neuroscience are linked to traits that people think about (optimism, personality etc) through surveys, so even if their methods were perfect, the results still wouldn't be.

On your survey point, I would agree that the participant pools are quite limited and generalizability is quite low, but your example is ludicrous. No one does surveys of gravity in psychology, surveys are carried out to investigate the manner in which people conceptualise their experience.

For a really great roundup of the problems with typical social science participant pools, see: Heinrich et al: humancond.org/_media/papers/weirdest_people.pdf

Indeed, the entire issue of BBS that that article appears in is well worth a look for a deeper understanding of these problems.

These surveys should then be calibrated against behavioural outcomes, but this does not happen often enough, which is a major issue in my view.

Thats the major problem with surveys, well that and some poor methods accepted too uncritically within the field (factor analysis).

To summate, psychology has many, many problems and I fully support this reproducibility effort (and it will expose a lot of findings as non-replicable). But don't single out surveys for derision, neuroscience deserves as much if not more of your scorn.

As a recent Psychology PhD, I agree completely, and I'll add this: people outside the field commonly believe that neuroimaging tools like fMRI are better and "harder" science than more traditional experiments that measure human behavior, but they have it totally wrong.

Here's an analogy for computer nerds: imagine if you had an "fMRI" of your computer's operation. You'd see that different tasks result in different parts of the computer "lighting up". Tasks such as graphics, disk I/O, numerical processing, etc., would lead to different patterns of activation. And if your scanner had could resolve, say, 0.1mm voxels in your CPU, you might even learn that certain parts of the CPU are related to certain tasks. But what all this tells you is something about the gross physical structure of the computer; it doesn't tell you much about the abstract, logical structure of the computer. To draw an analogy, you learn something about the brain, but not very much about the mind.

What's of interest to most people (except hardware engineers) is what the computer does, not what parts activate.

With a "computer fMRI", you'd learn little about how, say, a filesystem works, how a programming language works, or much of anything that's of interest at the functional level of the computer. The same is true of fMRI and humans: it doesn't tell us much about the mind works. Instead, it tells us that there's a brain region associated with some task. For example, one of the major recent-ish findings with fMRI is that there's a small brain region that's associated with face recognition.

I'm not saying that traditional experimental psychology is going to answer everything. The mind is very complicated and I'm pessimistic that we'll ever have a good reductive model, the way that we do in many other sciences. I'm also not saying that neuroimaging is totally useless -- but it's certainly not as enlightening as many people imagine.

Here's a summary of a paper that found that lay people thought that psychological explanations were more convincing with neuroscience talk (even though the "neuroscience" was totally irrelevant), and that those with experience in neuroscience/psychology thought the opposite: http://scienceblogs.com/cognitivedaily/2008/03/when_we_see_a...

Here's another one that did something similar, but with brain pictures: http://scienceblogs.com/cognitivedaily/2008/06/whats_more_co...

Add in that a lot of the psychology tradition is based on, let's say, "less than strictly scientific" traditions and personality cult.

http://en.wikipedia.org/wiki/Symbol_grounding

The objective "facts" provided by fMRI are meaningless on their own.

You do not use surveys to learn about gravity. That would be stupid. You use surveys whenever there is no better way to get the data you want. We have better ways to learn about gravity than surveys, so we use those. This is true for all scientific methods we have: All are not perfect (some are better than others, though) and all are not appropriate to answer every question.

(I know very little about psychology so I’m going to put on my social scientist hat.) The Large Hadron Collider can tell us nothing about which political issue people think is the most important one (one of the variables needed for Agenda Setting research, for example), so even though it’s a great tool to learn about gravity, it’s not a great tool to learn about Agenda Setting.

fMRIs are currently also about as useful as the LHC when it comes to answering questions like that. fMRIs are not very precise, we don’t understand the underlying system (brain!) thoroughly enough, they are expensive so we can’t look at very many people and have a representative sample.

They are just not good tools when it comes to answering certain questions and surveys become then our only realistic option, warts and all. (Some warts every good social scientist knows about: social desirability bias, reactivity, ceiling and floor effects, …)

So, Agenda Setting. The hypothesis is that issues about which the media reports a lot are also seen as important by the recipients. If you want to answer that question you have to do a quantitative content analysis and a survey. There is no other way to get to that data.

Surveys are not dubious tools. In the context of this research question, for example, an fMRI would be a much more dubious research tool.

Reply on news.ycombinator.com