Comment by rjf72 | Hacker News Reader

Comment by rjf72 | Hacker News Reader

The paper that this article is discussing was published in The Journal of Personality and Social Psychology. That's interesting for two reasons. The first is that it's a rather well regarded journal with a very high impact rate. The other is because it's one of the journals that's driving the ongoing replication crisis plaguing psychology. [1]

In particular the journal this was published in was found to have an overall replication rate of 23%. Another way of putting this is that if you assumed the exact opposite of everything you read that was published in that journal (e.g. - what they claim is statistically significant, is not actually significant) you'd be vastly more informed than a person who took what was published in the journal at face value.

Of course that does not mean that this study is 'false', but you should take it with the same degree of credulity that you'd take something published in a venue when you know 77% of what is published is false. I mention this largely because I think there's a substantial degree of dissonance between the crisis in psychology and people continuing to take studies in psychology at face value. It's illogical.

[1] - https://en.wikipedia.org/wiki/Replication_crisis#Psychology_...

> if you assumed the exact opposite of everything you read that was published in that journal ... you'd be vastly more informed

I don't think the opposite of not knowing is knowing (the opposite fact) :-)

That said, I agree with your main point on an even wider level. I read such very general headlines and conclusions - and when I read the actual paper there is a very specific and very limited experiment, with very specific people in a very specific context. I'm always left wondering how one gets from such a specific thing to such generalizations, especially when it's about such a complex subject as human behavior where tiny variations can produce radically different outcomes.

---

PS: Animats says it's paywalled, but the PDF on the linked page is accessible and has the data and details? About the N=150,949 part:

> EFL collected the data from small-business owners at the time they were applying for a loan at a microfinance institution (MFI) in Mexico from 2015 to 2017. As part of the loan application process, a prospective borrower had to complete a short psychometric assessment developed by EFL.

That's just what I said, it's very specific.

> The psychometric assessment that EFL developed consists of many modules, one of which is relevant for the purposes of the present investigation: the flashcard game. The flashcard game is a cognitive test on memory and executive functioning. In the flashcard game, participants are presented with an image; after pressing a key, they are shown a second image. Participants are then asked to indicate whether the second image matches the first. The flashcard game is scored based on whether or not the participant correctly identifies a match.

and

> At the end of the flashcard exercise, participants saw a question that asked them to estimate how well they think they did on this particular exercise relative to other applicants.... This question was added to the psychometric application for the purpose of this study

I don't know about others, but I have a hard time seeing such tests justifying the headlines.

I'm always left wondering how one gets from such a specific thing to such generalizations

In this case, I will suggest that someone is simply marrying general "common knowledge" to the study specifics.

Classism is very real. I've become very aware of its effects in recent years.

My mother's mother came from a low level noble family that sold the title. I grew up in a nice suburb, daughter of a retired decorated soldier. I was one of the top people in my high school, arguably one of the top people in my state academically, based on placing 3rd in a statewide competition and being a National Merit Scholarship winner.

In recent years, I've been quite poor and spent several years homeless. My social experience suggests that my recent poverty completely wipes out the value of my academic accomplishments etc in the eyes of a great many people.

It's been a huge problem for trying to get my life back. In the eyes of many people, I'm too poor to be considered competent enough to hire for anything.

Oh I don't want to doubt the result, I think it's probably true. I even posted this on my FB account (which I only use to bookmark and possibly spread "sciency" links I find interesting, not for anything personal) - but then deleted it again. Because I have to admit as much as I personally like the result because it confirms my personal believes, I find the foundation the headline is based on shaky ground given that it's supposed to be science and not opinion. I can't complain about others if I find my own bias influences whether I ignore how results are obtained and just spread the headline hoping (and knowing) that other people (following my news feed) won't read the details.

I'm pretty socially observant and my adult sons are rather aspie. In raising them, I had to really up my game for trying to find data and explanations for social things.

I once had a conversation with my oldest where I tried to say something like "anger is like fire" as a metaphor and he leapt to some conclusion like "so, you rub two people together and they explode?"

Um, no. Forget that metaphor. Let me get back to you.

So we spend a lot of time talking about both social things and how to find actually good information and data because psych studies are so often terribly done and they are inherently hard to do well because most studies are de facto a form of manipulation, which itself pollutes the data.

I think few people would disagree with you. But this is really the biggest danger. If you look at the studies that have been let's kindly say 'found to be less than well supported', you'd also find many that you would intuitively agree with the outcome of, or at least want to agree with. Psychology, and social psychology in particular, has a habit of confirming our biases and making us feel good. This, in turn, takes us off our guard for bad science. And by "us", I think this is something that's likely even issue within the field itself. You don't get a 23% replication rate without there being some very severe problems within a field.

Take for contrast, astrology. We all now know it's obviously fake. It even relies upon things we know are wrong, such as assuming planets can magically stop and start going the other way in their orbital trajectory. That assumption was based on an optical illusion driven by the assumption that everything revolved around the Earth. But for hundreds of years astrology was treated as respectable a science as any other. And I think it was for a similar reason. So compelling and entertaining are the correlations for astrology that even today you have millions, if not billions, of people that still partake in it to varying degrees.

The psychology replication crisis is, in my opinion, hurting science as a whole. We should not indulge bad science even if it happens to tell us what we want to hear. It's making a mockery of what science is truly about.

I think few people would disagree with you. But this is really the biggest danger.

I was merely describing how such happens, not defending the process.

Take for contrast, astrology. We all now know it's obviously fake.

You happen to be speaking to someone who has studied astrology in earnest and believes it has some merit. Astrology is not immune to the rubric "90% of everything is crap." "Sun sign" or "popular" astrology is often quite badly done, but serious astrology is highly mathematical and the practice of marking retrogrades is a perfectly accurate assessment of where those planets appear in relation to the earth because it is a geocentric model, having nothing whatsoever to do with a scientific belief in a geocentric model rather than a solar system.

Astrology is part of the spiritual beliefs of many people. It wouldn't be okay to casually suggest that Christians are nutters for believing Jesus rose from the dead.

The paper looks more interesting, but it's paywalled.[1] The study involved over 150,000 people, which is impressive. Too many psych studies get subjects from undergrad students, or worse, Mechanical Turk.

Here's the first author's web page.[2] He seems to specialize in this area. Unfortunately, none of his papers listed link to the full text.

[1] https://psycnet.apa.org/doiLanding?doi=10.1037%2Fpspi0000187

[2] https://www.darden.virginia.edu/faculty-research/directory/p...

https://sci-hub.tw/#about for bypassing paywalls.

But I would not expect to find any huge red flag. This journal, in spite of having an absolutely abysmal replication rate, is one of the leading journals in psychology. It's not like the 77% of studies that can't be replicated were just obviously broken in some way. It's simply that when people try to do the same thing, the results do not hold. Though, in this case at least a peer comment provided some more insight into the 150k.

Reply on news.ycombinator.com