Hacker News Reader

Comment by colechristensen | Hacker News Reader

There's a problem I have with this kind of science.

It boils down to "We tried to achieve X by doing Y and failed, therefore X isn't possible".

Assuming Y is the correct way to do X and then giving up. Instead it should almost be something like a competition. "We tried to achieve X by doing Y_{1..n} and these n had statistically significant effects of these sizes"

When you have a lot of people trying to accomplish something, some of them will end up being successful if the thing is possible, but there is an art to finding that optimum and published science really often seems to fall short.

View on HN

I think you are mischaracterizing this meta-analysis. First, a meta-analysis is never about just trying one thing; that's what a single study is for. A meta-analysis looks at many studies, over a long period of time and tries to find consistent patterns that emerge from the whole. That's how we try to systematically figure out if results are by chance, or if there's a real effect. The pattern these authors are pulling from all of these studies is there is no real effect. When you find no real effect over many studies for a long time, you inevitably have to confront the question: is what we're trying to do possible?

> When you find no real effect over many studies for a long time, you inevitably have to confront the question: is what we're trying to do possible?

Devil's advocate: The same would apply to say, Alzheimer's treatments, where we've had no real effect over many studies for a long time. Should we give up?

Lack of progress is a good reason to backtrack up the search tree. Which is exactly what we are seeing with alzheimer’s research. Amyloid hypothesis isn’t working, so backtrack. It’s a logical error to extend that to “give up”

The logical extension of this would be that perhaps it's time to not spend as much effort on some approaches at cognitive training, but not to give up on possibility of the entire idea.

>When you find no real effect over many studies for a long time, you inevitably have to confront the question: is what we're trying to do possible?

This is what I'm criticizing. The methodology for searching for real effects is broken.

one problem with meta analyses is that they often collapse studies accross theoretically important details, and thus unsurprisingly find weak or inconsistent effects. I saw this in a lot of metaanalyses of memory function.

We also have Q P S R T U V and W that need funding and aren't getting it because X is getting all of it.

If X is too hard try something else that's easier.

Reply on news.ycombinator.com