Social Psychology Is A Flamethrower
slatestarcodex.com
slatestarcodex.com
The first thing I learned is that my friend actually took part in the survey by "putting herself in the mindframe of a 13-year-old and filling it in as the kid would". After a short discussion about how doing so turns the whole thesis into pile of nonsense, she (herself a graduate from a liberal arts college) told me that the survey author was actually one of the better student in that she actually made the survey - as most people fill the surveys themselves or just make the data up.
So things like these give me trust issues with all the "soft sciences". I do consider most of the sociology nonsense, because I seriously doubt that people who make up data for their thesis will suddenly turn into honest researchers after graduation. Yes, the "best research in social psychology" might be "as well-supported as anything in physics or biology", but I doubt it's more than a small part of research done, and mostly research from first principles (so there isn't much chance that the research you're basing your paper on is crap).
The 'hard' sciences in the US are developing as much of a credibility problem as its 'soft' sciences. Institutional science may not be working out for us as well as has been hoped. It's a recent establishment and may not survive without a reversal in the decay of standards.
You have fallen victim to the Typical Mind fallacy. Most undergraduate students are there for the paper that will make it easier to get a good job. Most people are not like you.
More than half of the graduate business students surveyed recently admitted to cheating at least once during the last academic year, according to a report released on Monday. The report, "Academic Dishonesty in Graduate Business Programs: Prevalence, Causes, and Proposed Action," is based on survey responses from 5,331 students at 32 graduate schools in the United States and Canada, and is scheduled for publication this month in Academy of Management Learning & Education. The survey found that 56 percent of graduate business students -- most of whom are pursuing M.B.A.'s -- had cheated, compared with 47 percent of graduate students in nonbusiness programs.
Please note almost half of nonbusiness GRAD programs cheated. It is going to be way, way higher in undergrad.
Key words: in the US. You can't generalize from one institution in one country to all institutions on the planet.
I went to the NSF grant database and searched for "astrophysics":
http://www.nsf.gov/awardsearch/simpleSearchResult?queryText=...
> Result 1. Deliverable: a terabyte-scale real-time data exchange and correlation platform. Ultimate purpose: expediate collection & analysis process for data sourced from multiple observatories.
> Result 2. Deliverable: "the coordinated observation of nearby supernovae with optical and near-infrared spectrographs on 4m and 8m class telescope..." Ultimate Purpose: The study of thermonuclear supernovae ... [which] have played a central role in the discovery of the acceleration of the expansion of the universe and have a key role in attempts to constrain the nature of dark energy.
> Result 3. Deliverable: "an underground accelerator laboratory" Ultimate Purpose: " address three long-standing fundamental problems in nuclear astrophysics: solar neutrino sources and the core metallicity of the sun; carbon-based nucleosynthesis; and neutron sources for the production of trans-iron elements in stars."
> ... that's all I care to summarize
Where is the tortuous logic and bullshit justification you keep going on about? It looks to me like the grants are going towards funding legitimate scientific inquiries into legitimate scientific questions.
Thankfully, I don't judge you entirely on it ;)
Could you please elaborate a little bit on what you consider stupid in it? I'm always eager to learn :).
Plus rather than saying undergraduate thesis say assignment. Because that's what it really is, not at all related to peer review papers or a doctorate.
Of course my comment is partly tongue in cheek, given that its form necessarily shadows your own statement for my desired level of irony.
My own biases come in part from my GF. She's an economist - a social science - and has opened my eyes in many respects. Most of all, it's extremely hard to get good data in the social sciences. She works in healthcare policy evaluation, but all the people collecting data are not trained in it - they're filling out forms, themselves often designed by people not trained in it either. And of course she runs up against the usual "you have to do it for the children" people who dismiss any kind of rational analysis of the cost / benefit of various different healthcare interventions.
Anyway, I'm a bit sensitive about this, because I see a lot of arrogant ignorance, especially in IT, with guys (usually guys in our field) dismissing whole fields that very clever men and women have dedicated entire lives to, without bothering to learn anything about what they're dismissing.
Bad data is easy to get.
It is also standard practice to disguise these kinds of filter questions in surveys and to leave possible respondents in the dark about the ultimate purpose of the survey (insofar as that is possible and it isn’t ethically necessary to be transparent about certain aspects). That way respondents outside the filter criteria hopefully couldn’t participate even if they wanted to fake their way in. The way I do it, no one besides my advisor and a limited pool of pre-testers (who I make sure to tell not to participate in the main survey – and I also try to keep the link to the survey away from them) knows what the survey is about and what are the filter criteria.
That’s students. Most of them won’t go on to do any actual worthwhile research that will ever be cited by anyone. I would hope that the standards are even higher for research that’s published in journals.
Social research is tough even if everyone behaves ethically. No doubt about that. But I have never encountered any culture of fraudulent behaviour outside of lazy students who will never publish anything anyone cares for.
FWIW I'm an undergrad who is currently writing his thesis (empirical economics). I use the German Socio Economic Panel data set, which is a representative survey of ~11,000 German households (totaling about 79,000 individuals over 28 years). The reason why I picked a data set instead of collecting my own data is precisely to avoid the situation of the social sciences student you describe. As an undergrad, getting data is extremely difficult. You usually get about 6 weeks to write your thesis and you have absolutely no training in data collection and/or survey design. Getting data from students is your best bet, but nobody wants to answer your questionnaire because…hold your breath…everyone is sending out questionnaires. I get spammed with these things. Luckily, nobody has asked me to pretend that I am a 13 year old. So quite frankly most students panic after about 2-3 weeks and they will try to do anything to get out the situation. The alternative is not graduating and paying tuition for another year.
Secondly: Undergrad thesis =\= scientific paper. The fraction of undergrads who end up being social scientists is…tiny. Out of 300-400 econ students who graduate at my uni every year, I think 20 or less will go onto grad school (I'm proxying the number by thinking about how many went to do an MPhil). Now econ grad school is a different beast. Essentially you either end up doing experimental research or you work with observational data (assuming you are not a theoretician, those exist too; Economics is pretty much applied mathematics at that level.). Experiments can be checked and working with observational data requires a trove of econometrics. Trying to fool a panel of econometricians isn't easy, especially when these days you have to send in your data to the journal (or so my supervisor says; he has a paper forthcoming in the Quarterly Journal of Economics).
Social science in general is becoming more and more computational and statistically rigorous. Gone are the days when an "expert" would find a correlation between two variables and then speculate which way the causality flows…if any.
As an aside, I first misread your post and thought you were describing the process of taking your own survey, while putting your self in the mind-frame of the person filling it out, in order to imagine what goes through someone's head when they fill it in. This seems like an extremely good idea.
Science starts at dissertations.
I hear "arguments" like this all the time against data-mining public information and crowd sourcing and making it open on our site (and through an API) through our feedback and they make me chuckle seeing how they had to engage and co-opt a multitude of psychological/behavioral theories not drafted by themselves to say such things. Even more funnier is when the things they are protesting has been done in "private" for a very long time. Kind of reminds me of how the whole Snowden thing plays out on HN and in the general culture at large, despite the mumblings of such since the very beginning… quite amusing to say the least…
Ah, yes, quite droll that people who don't make money from the thing I make money from have different moral opinions on it. Very amusing indeed. Because, of course, they're obviously wrong.
And they use other people's words when arguing their point? I'd point out that this is how just how ideas pass through society, but I'll settle for laughing at the fact you see unaware that you're also doing exactly that in your post.
I guess those who think they are morally right can just give themselves a big round of applause while they continue to engage in behaviors that act against such moral beliefs, whether they are willing to admit that to themselves or not and I'll just keep exploring the infinite. :D
Nope.
The best research in social psychology is as well-supported as anything in physics or biology, and much more intuitively comprehensible.
And then he follows it up with:
Social psychology experiments in the laboratory tend to throw up spectacular mind-boggling effects. Many of these fail to replicate and are later discredited. The ones that do replicate are not always generalizable – sometimes an even slightly different situation will remove the effect or create exactly the opposite effect. The effects that remain robust in the laboratory may be too short-lasting or too specific to have any importance in real life. And the ones that do matter in real life may respond unpredictably or even paradoxically to attempts to control them.
The behaviour often have nothing to do with reality - when measured, everything tends towards the ideal.
And the experiments can often result in the results you want if the double blinds are removed.
I always fall back onto the Survey from "Yes Minister" to explain how that works
You mean, 'non double blind' experiments?
Generator-based tests (as in Haskell's Quickcheck) could help here. Standard unit tests run assertions with hardcoded input values such as [0, 1, -1]. Generator based tests instead try to show that properties hold over randomly generated inputs, by replacing 1 with a generator of positive integers, -1 with a generator of negative integers, etc.
Imagine if social psychologists ran generator-based studies as a hedge against unknown unknowns. The biggest challenge would be generating and executing valid permutations of the original experiment. Perhaps ad network AB-testing could be repurposed to run experiments as ads with randomly generated values.
there is a lot of research done with surveys, and most of these surveys have an inherent assumption: that the people live on an island and are not autocorrelated.
so he created a false network of people with random fake political views, etc and added network autocorrelation. they found that random views would cluster with demographics and were very statistically significant - but the views were random.
in a similar experiment, a music sharing website to collect data on its users separated users into 'worlds'. users could see what songs were downloaded by other users within the world. and each world had its own popular songs despite being made up of random people.
problem with networks (people) is that even a few missing nodes makes analyzing the network almost useless.
I don't understand the proposed alternative. There is a widespread disconnect between the ethical view of consent ("enthusiastic yes") and the equally common disregard of consent ("well, she only said no the first time but then got real quiet...seemed OK"). Families and friend circles often fail to support rape victims or confront rapists. We're facing a structural cultural problem, and the solution is to...not talk about it?
Second, you can explain proper consent without lecturing on issues like lax enforcement and prevalence of rape.
Telling men that "most men think drunk consent is not consent" would be in line with this, and is I think what the article is talking about.
I've never understood this attitude:
"What you are doing is costly and has been shown to be useless/counterproductive."
"But we have to do SOMETHING."
Besides, the author was not saying we shouldn't talk about the problem, just that this one particular way of talking about it worsens the problem.
Issues i have:
1.) What would the practical consequences be if social psychology and it's knowledge would be erased. Not many that i can think of. Considering that the area has immense potential applications this is pretty telling.
2.) What have been the most important research breakthroughs of the last 20-30 years? There are new fads for sure (behavioral economics, neuroeconomics and evolutionary psych i look at you) but they all coexist peacefully. No falsification is happening, just new profs need new theories for publication.
3.) There is NO theoretical rigor. If you do have tons of theories you would suppose that there is much theoretic work going on clarifying and contrasting this theories to enable empirical tests of their validity. Not here. The more theories the merrier. Why? Every researcher needs his own theory or not crowded field so he doesn't rock the boat. That has lead to the development that one experiment is done by two grad students and a significant result in one direction is interpreted by one theory and in the other direction by another theory. Basically you give a sample of students a questionnaire and if it goes one way it proves that XY gave an evolutionary advantage to the prehistoric people if it went the other way it proves that the utility curves cringle on the right side. Both ways the researcher has something to publish. I do think that the 2 theories per experiment limitation is rather arbitrary and inefficient tough. There is definite room for improvement here.
4.) Many recent work try to address this by going theory free. All this different effects are the result. This is also practical because there is always room for more effects to research and if you happen to need one to explain a particular fact then there are always a ton to choose from.
5.) Cargo cult use of statistics. You could make an argument that most social science researchers are not able to understand the whys and hows of statistics and decision theory so giving them recipes to follow is better than the alternative. Might be. It just does not work. It really does not. Also the imperative of the researcher is not to generate knowledge but publishable significant results.
Basically i do think that we have to start over and scrap the work done so far. And no, i do not have a better alternative at hand. That is not necessary to see that the current process is broken and produces nothing useful and binds intellectual potential, tough.
Much like macroeconomics, the problem with falsifying our hypotheses in the field is that you can't just build a culture in the laboratory. Even worse, there's no such thing as a control group: everything is observing everything and trying to self-modify to copy the good ideas of others, all the time.
There's no link to the research.
Does anyone have any links? I'd be interested to know of this is just more lack of reporting of crime in a country that doesn't do much about that crime.
(This is broadly similar to other correlations you may've heard about with more regular porn: porn seems to substitute for sex crimes, and not increase crimes like rape.)
This is a very naive view of the idea that violent movies increase violence in the population - the author is acting a little surprised that the results actually show that crime goes down a little on opening weekend and doesn't change from normal in the weeks before and after. It makes no long-term analysis of the baseline; the culture of having violent movies so prominent affecting levels of violence in the population. It just shows that violent people like to go to violent movies, not that the baseline violence is increased or decreased from those movies.
Reading over it again, the author is just guilty of bad journalism. The article he's quoting for this section is "Does opening weekend of violent films have an effect on violence rates?", which is a really very specific view, but is painted by the author as an incredibly broad "Violence In The Media Prevents Violent Crime". The irony is that it's the very first example given after a screed about how sloppy the field is.
Look, I'm not saying that social psych research couldn't do with improvement in quality, but that someone who's pointing fingers should make sure their own stuff is tight.
Personally, I'd think it would be hard to directly test the proposition "existence of violent movies at all increases baseline level of violence in society". You can't do a randomized controlled trial, and there are no good natural experiments. Trying to measure both variables across a range of communities would be hopelessly polluted by confounders. Therefore, I'd spect neither side to have a very strong case, which is exactly the point of the article.
No, my comparison to the climate science debate was more about using a selected study and demanding it has parity with the collection of whatever evidence is on 'the other side'. This is what the author is doing in the conclusion; suggesting that because he's found one or two studies that go against the grain, that the side those studies oppose are equally poorly researched.
Yes, it is difficult to measure, I absolutely agree. But the author could still have made his point without resorting to spin-doctoring. He is suggesting that the line of reasoning is valid and plausible - and it's not, because he's heavily misrepresenting the data in that first case. If that was his point - that some studies heavily misrepresent the data like that - then he should have exposed studies instead of repeating the process as a counterpoint.
In any case, this is a line from his conclusion: the laboratory experiments that experimental exposure to violence causes people to play contrived games in a more aggressive manner couldn’t catch that in the real world, violent movies decrease crime. He's taken this blip from blockbuster opening weekends and taken it as gospel in his rationale. "Because violent people like to go see a movie with everyone else, and they don't do more crimes later on to 'catch up', violent movies do not increase violent behaviour". It's a really naive view of the argument around violent movies and violence in the public, that it only has short-term effects measured in weeks, yet he's using it in his conclusion (not just in his 'case studies').
By "couldn’t catch that in the real world, violent movies decrease crime", the author has clearly taken that faulty interpretation of the study as an overarching truth - even after saying that both sides of the story are 'plausible and intuitive', he's still declared that 'what happens in the real world' is based on the misrepresentation of his study. I don't think the spindoctor-esque misrepresentation was a conscious decision on the author's part, else it would have been highlighted more clearly. I think it was just a mistake.