Bad science as genre fiction | Hacker News Reader

Bad science as genre fiction

statmodeling.stat.columbia.edu

29 pointsnabla910 comments

I was going to write that this comparison makes me think of largely-wrong “surprising truths!” airport non-fic books, but I see on further reading that the author is way ahead of me and this analysis includes those in its scope.

To the formulaic-ness of it, there’s the recurring ha-ha-only-serious quip from the If Books Could Kill podcasters that the more bad self-help, political hack, and trendy “science” books they read, the more sure they are that they’re all actually the same book.

100%

I used to be a sucker for this type of books: non-fiction, "science-based", feel-good, insight-porn pamphlets inflated to 300 pages with shoehorned anecdotes and a very distinct flavour of extrapolative slop.

They are usually written by journalists, TED-addicted researchers and, more recently, Youtube and Twitter influencers.

The depressing thing is: For me, the realization that 99% of them are shit only came after reading hundreds of them. My guess is that my brain needed enough samples in order to start accurately classifying them.

It's like one develops taste only after being exposed to a lot of terrible art, since then the actual great art stands out like a beacon.

In Adler’s (How to Read a Book) terms, I think lots of folks don’t naturally learn to do his “third reading”[1] simultaneously with the first two readings (or maybe at all?), and tend to read a book only once, or to read it more than once but doing the “same” reading each time. That plus some assumptions that the social proof of a book getting recommended by, say, Bill Gates and being a bestseller means it must not just be a whole bunch of lazy crap that sometimes contradicts its own citations, can explain a lot of things slipping by a person that, viewed with hindsight, were pretty sketchy.

Certain social science or liberal arts programs are pretty good at, if nothing else, pushing students to get better at that “third reading” and developing a heuristic, automatic sense for when a piece is behaving in a maybe-dodgy way, but I think a lot of other programs aren’t, so even educated people, at least those who don’t just naturally pick the skill up young, may take a rougher road to it, if they ever pick it up. (Yes, I’m aware that my calling out some social science training as a good way to hone that skill is funny, given how much junk science, pop and otherwise, centers around the social sciences)

[1] Adler’s three readings are: 1) coming to terms—that is, seeking an understanding of the basic assumptions, definitions, context, et c, the author is operating under, and the basic thrust or topic or goal of the work; 2) Understanding the author’s argument, message, points and so forth; 3) critical reading—is the message any good? Does it hold together? Are the definitions and premises well-considered, useful, and applied consistently? Does that passage there look a bit iffy, is perhaps worded like the author’s trying to get away with something, and maybe needs a closer look? If such-and-such part crumbles on closer examination or consultation of other sources, what of the rest is salvageable, if any?

The junk science from social sciences is a combination of students/researchers who both lack the ability to read critically and are given work that requires it fundamentally. Any real critical sociologist is going to be far more scientific than most scientists when it comes to their analysis since they have both the rigour of their method and the eye of self-reflexive objectivity, but you don't always get those people. Mostly not, most academics are just fairly mediocre people who were smart enough to get through their classes, kissed enough ass, are seemingly competent and lucky to land a good position.

Basically none of the books upper thread talks about was written by an actual scientist trained in the area they write about. They were written by popular people, either not scientists or writing outside of their expertise.

If only. The replication crisis means a lot of scientists have written books about their area based on bad research. Not only they were trained in their area, some of them even invented it.

I'll give you one concrete example: "Presence: Bringing Your Boldest Self to Your Biggest Challenges", by Amy Cuddy. That book was based on the (now completely falsified) theory of "Power posing"[0], which the author published peer-reviewed papers on.

It's a neat little grift: publish a couple of papers, get yourself a TED talk, sign a book deal, and suddenly you're getting hired to teach leadership courses to clueless MBAs and give conference talks to bored executives.

[0]: https://en.wikipedia.org/wiki/Power_posing

I tend to avoid stuff by journalists and prefer stuff by researchers writing about their own field for this reason. Journalists often seek narrarive.

Researchers can still be wrong, but usually less belligerently so. The academic publication process and the science it prodices, for all its flaws, generally forces researchers towards more nuanced and continent positions.

Yeah, lately I've begun asking myself "How much work would you actually need to do to know this or prove this, and what would that work look like"

I believe that we process facts in a narrative. If you look at a bare fact, your mind automatically creates a narrative.

Murder rate 1%

22 Angstrom

Purple Burglar Alarm

That narrative takes place in a framing story.

'Science', among other things, is a very detailed framing story that modern society uses to evaluate facts.

An individual's framing story includes all of their history and biases and motivated reasoning and incentives and all the rest of the fallacies.

So yes, bad science is fiction, but even good science is narrative. If this feels wrong, go re-read a paper. Look at a figure or a sentence in the abstract. Your mind constructs a narrative.

Andrew addressed this in the comments:

  Regarding storytelling: Thomas Basbøll and I have argued that stories are important not just for communicating science to others but also in doing good science.

 My perspective here is:

  1. Fictional stories are an informal sort of predictive model checking, in which the process of working out a story is a way of understanding the implications of a scenario.

  2. Nonfiction stories are a sort of prior predictive check, where the unexpected twist that makes the story so satisfying represents a revelation of some aspect of reality that confounds our expectations, that is, does not fit with our implicit or explicit model of the world.

That second point is a bit murky but this is a comment on a blog :) Here is the much more detailed paper he co-wrote with Basbøll: http://stat.columbia.edu/~gelman/research/published/storytel...

Reply on news.ycombinator.com