A study on dishonesty was based on fraudulent data | Hacker News Reader

A study on dishonesty was based on fraudulent data

economist.com

114 pointsbreitling42 comments

Open articleView on HN

Recent discussions:

https://news.ycombinator.com/item?id=28210642 (51 comments)

https://news.ycombinator.com/item?id=28257860 (90 comments)

https://news.ycombinator.com/item?id=28264097 (104 comments)

Thanks! Macroexpanded:

Noted study in psychology fails to replicate, crumbles with evidence of fraud - https://news.ycombinator.com/item?id=28264097 - Aug 2021 (102 comments)

A Big Study About Honesty Turns Out to Be Based on Fake Data - https://news.ycombinator.com/item?id=28257860 - Aug 2021 (88 comments)

Evidence of fraud in an influential field experiment about dishonesty - https://news.ycombinator.com/item?id=28210642 - Aug 2021 (51 comments)

What's interesting is that Ariely is the chief behavioral officer to Lemonade (LMND) an insurtech company https://www.lemonade.com/blog/oh-behave/

This is Dan Ariely's response to the case:

http://datacolada.org/storage_strong/DanBlogComment_Aug_16_2...

This is very helpful. Assuming Dan is being honest, it seems pretty obvious the insurance company is at fault here. It's so hard to know though. He could be covering for the whole group, or a teammate, or even himself.

Frankly I'm fine believing it was the insurance company though. Auto insurance has already screwed me before and now I have an irrational and emotionally charged inclination to view them as villain

However, car insurance company has nothing to gain from faking the numbers. Researches who publish an influential study that gets a lot of citations, and because of it they can get tenure, big contracts or lucrative job offers, have everything to gain from faking numbers.

That's a good point, but it could be that the person tasked with gathering the numbers had something to gain (their own time, if they felt they could fudge and still get paid), and nobody checked the work. Still seems like a long shot. Typically we ask, "Cui bono?"

The guy who’s now a senior executive at a very large public company making quite good money and was very well regarded in the field for a long time for making a breakthrough discovery?

This is only true of researchers whose career depends on fraud.

In this case, if anything, an established researcher has everything to lose from such data, whereas the insurance company could give out flaky data for whatever reason and not care about it.

Yeah… in cases like this it always seems to be the PI doing something shady. Notwithstanding your personal hatred for the insurance industry, I would bias my prior towards malfeasance on the PIs part, especially since his defense is basically the equivalent “I totally have a girlfriend, she just goes to a different school… in Canada. You don’t know her”. This is a smart CYA response and nothing more. When you hear the totality of the evidence (primarily that he kept promoting this study after he 100% knew it didn’t replicate, he was investigating other similar kinds of effects that would have to be dead ends), it paints a clearer picture of him as a researcher.

The thing is Dan has a history of lying and exaggerating results, specifically around this idea of nudging peoples behaviors.

https://www.buzzfeednews.com/article/stephaniemlee/dan-ariel...

While it does seem like the insurance company is most likely "at fault" for the fraudulent data, how did none of the researchers catch this?

My understanding is that even looking at a histogram of the mileages should have raised serious doubts about the data.

> how did none of the researchers catch this

That's so, so easy to happen, specially 10 years ago. If they're not already primed to think a dataset might be fraudulent, a researcher might likely just think they've won the statistical signal lottery.

Having had to come up with simulated datasets before, whoever faked this data clearly had no understanding of how the random number generator in excel/elsewhere would work.

It is EXTREMELY easy to generate a normally distributed dataset, with a zero lower bound. Literally one line of code/formulas in excel e.g., =max(NORMINV(RAND(),10000,5000),0). We learned it in the first three weeks of my (graduate) intro stats class. You could then apply that formula to however many data points you needed, and in this case pass in a well known mean for # of miles driven per year.

It's been 6 years since I took that class and I still remember the formula. I suppose it's possible that Ariely, a professor with decades of years of experience as a researcher, who had to have taken more stats classes than me (literally one) to get his PhD, would be dumb enough to use a random # generator.

But that is an awfully, awfully stats illiterate mistake to make.

Certainly, his team would have something to gain by a fascinating finding, though at this point Ariely already had two massively successful books prior to this finding. But the analyst at XYZ insurance that didnt want to be bothered to pull an actual dataset from SQL or oracle also couldve just been lazy...

Meanwhile, a study on irony was found to be based on allegory.

But that was just a coincidence.

> “I did not fabricate the data,” he insists. “I am willing to do a lie detection test on that.”

I imagine a world-famous psychologist probably knows how to beat a lie detector test. Not that it proves he's lying, just that this statement doesn't mean much.

It's another layer of humor to this thing, that the proposal is to resolve the issue with a phrenology-ouija board type of thing, known to be incredibly unreliable. Why would anyone even mention a lie detector?

They're only unreliable when administered to laypeople. When administered to psychologists, polygraphs reliably prove them to be truth-tellers!

Not surprising that a psuedoscientist wants to take a psuedoscientific test.

Ah, come on.

They just wanted to test if anyone notices. A meta-experiment on the Replication Crisis, so to speak.

https://en.wikipedia.org/wiki/Replication_crisis

Or some Excel BS.

As I already said elsewhere:

Nothing is true, everything is Excel...

http://web.archive.org/web/20160827085658/https://www.washin...

Just wait for his next book...

Intuitive argument against suggestions that Ariely might have faked the data... It seems unlikely to me that someone of Ariely's mathematical knowledge and research experience would fake data in such an obvious manner.

(Full disclosure: I've met with Ariely a couple times, 20 years ago, about cross-disciplinary research, and have a favorable impression of his character.)

This seems like a "It's too obvious" defense. I tend to think that very obvious explanations are usually right. Besides, contrary to the fraud being too obvious, it went undetected for a decade.

From what I can tell the data was faked by Ariely or the insurance company. Why would the insurance company fake data to support Ariely's hypothesis? What do they gain from that? Presumably, the insurance company is trying to find interventions to make customers behave more honestly - faking data to get spurious answers seems counter productive.

Ariely on the other hand has clear benefits from faking the data. He has the means, motive, and opportunity. On top of that, there's no other plausible suspect.

What sounds obvious (from how it was described in one of the articles) is that the data is fake. And that someone fabricated data that anyone who hadn't slept through the first day of Stats 101 would understand is fake data.

If I'm being creative for a moment, I can totally imagine a few other plausible scenarios, which also seem more likely than Ariely knowingly publishing obviously fake data.

I'm not going to mention those scenarios, because, in the kind of court of public opinion/speculation like we're seeing even in this very HN post, people are effectively convicted and executed upon speculation.

These are real people, consider they might be innocent, and therefore treat them as we'd like to be treated, were we innocent yet suddenly fell into the crosshairs of the Twittersphere loose canons.

https://archive.is/kSZmX

I'm a huge believer in letting the chips fall, but let's see what investigations yield prior to castigating any individual.

Now personally, reading about Dan's career, his writing, his talks, and even knowing one of his friends from the army, I fully believe, and want to continue to believe that he's innocent. His research, and his writing have proven fundamental to my world view in so many ways. He's frequently helped change how I think about things, approach problems, and generally view possibilities.

That however is anecdotal. Dan would be the first to say let's look at the data. So, let's wait, and look at the data.

Dan Ariely's reputation is now in tatters.

Is it? My wife was studying in a lab (real science, not social science) and there were only a handful of labs in the world working on the problem. One of the labs had a key paper that no one could ever replicate and it was widely understood to be fraud. The drama was even written up in one of the blogs focused on these issues. The PI to this day claims it’s all true and nothing was faked, still respected in their field. Every grad student working on that protein knows that result and the subsequent papers published by the same lab based on that result are complete lies.

Mind sharing which paper/PI you’re talking about?

My wife is asleep now but I set a reminder to ask her tomorrow. I believe it was this one https://retractionwatch.com/2018/03/23/caught-our-notice-the...

Posted in 2018 but people knew about it for years. I will try to get all the details as the link is fairly dry but basically they are working on this protein and this paper and the science that followed set back many grad students wasting time and money trying to create more science based on the fake science.

Hmm, the conclusion that "mitochondrial ferredoxin is required for assembly of cytosolic iron sulfur proteins" does seem to hold up though. Unless I'm mistaken.

My wife says that this is complicated and there is a lot of nuance, but there are specifics of their work that cannot be replicated, and in fact my wife's lab came to opposite conclusions for some of the results. So the labs that do work on this pathway simply ignore or refute in their future papers the Lill lab results that conflict. She said that whole paper is ignored and there was a meeting in Germany with the labs that work on this pathway and all the grad students agreed they did not trust work coming out of the Lill lab. She said she thought the paper was fully retracted and was surprised when I sent her the link that it wasn't fully retracted.

Interesting. I read his book Predictably Irrational and quite liked it (of course that doesn't imply he's a good scientist). Are there any other scandals he is involved in?

I was also wondering, because there were some other vague mentions, but couldn’t find anything concrete.

The only thing[0] I could find was one person who questioned how he could technically make a shredder do what he said he did in one experiment (and there were even photos of the shredded-but-only-at-the-edges papers)

[0] https://fraudbytes.blogspot.com/2021/08/top-honesty-research...

Interesting hearing Dan Ariely explain how he broke the shredder: https://drive.google.com/file/d/1ZTIm9nmurRm96kWKwxrl5dI3t7e...

Anybody with a good talent for picking up liars from a voice recording willing to share an opinion?

They contacted him and he apparently claimed that he broke the teeth on a shredder with a screwdriver. The teeth on shredders exist only to make "cross cuts". The previous generation of shredders did not have them, and cut the paper into long strips instead of confetti; which is fine for most purposes. So a claim that removing teeth prevents shredding is bogus.

(Edited to add) thinking about it, this could be an honest mistake. You could "remove the teeth with a screwdriver" by disassembling the shredder and removing some blades from the axle, which corresponds to how the shredder in this paper was supposed to work. If Dan Ariely got a grad student to modify the shredder, he could simply have misunderstood their explanation of how they did so. How the shredder was modified was not, after all, important to the experiment.

(Edit 2) Although, those who ran the experiment would need to keep the missing teeth out of the line of sight of the subjects.

It doesn't help his cause that he continued to promote this study's findings even though he knew they didn't replicate.

[0] https://statmodeling.stat.columbia.edu/2021/08/19/a-scandal-...

yeah that’s very damning, but still roughly the same scandal? ;-)

I think the comments on this page vaguely mention other things, but couldn’t find much concrete last I checked

But seems like it was a rational play from him. He made tons of money from his book and speaking engagements, and is not going to go to jail over this. He might be laughing all the way to the bank

Don’t know why you’re downvoted. He had every incentive to lie and has historically shown dubious judgments and hyperbole.

https://www.buzzfeednews.com/article/stephaniemlee/dan-ariel...

The things that jumps out to me is how statistics and studies like this shape the larger cultural discussions, legal discussions, and public policy discussions. It really shows how data can be munged, cherry picked, and altered to fit any narrative you want to send. Luckily over time scientific method should correct this but these can do serious damage in the short term.

You mean diligent humans can correct this. The Scientific Method is but a tool. One used during the fraudulent phase of research as much as during the detection one.

Reply on news.ycombinator.com