Comment by rossdavidh | Hacker News Reader

Comment by rossdavidh | Hacker News Reader

Parent Full thread

So, I agree it's an interesting idea, and I think this small study could hopefully spur more conclusive ones, but...

1) N=45

2) based on "semi-structured phone interview"

Neither of these is a deal-breaker, but what you'd really prefer is some sort of objective measure of sleep quantity, and how it impacted the person. Teenagers are, by and large, entirely aware that when adults ask them questions, they are likely to draw conclusions about policy towards teenagers. They want to sleep later. How likely is it that they would admit it, if they didn't really get any more sleep out of this?

Not saying it's not true, or that it isn't an interesting idea to try to put this "natural experiment" to work somehow, but I'm not sure this is really enough to draw any conclusions, and I actually already agreed with their conclusion before seeing this study.

A better method might be to compare home-school students to public schools, or compare summer-break to school year, and to use some more objective metric than a phone interview.

Agreed with all points. I often find that these sorts of observational studies with weak methods that provide evidence for something the unblinded study participants would want to support (e.g., this study, observational studies of the effect of reduced workweek hours on productivity, many of the observational basic income studies) are really hard to evaluate fairly. It becomes easy to latch onto the conclusions when the intervention is one you're predisposed to believe in and to latch onto the methodological weaknesses when it's one you start of opposed towards, and so I feel like they don't really advance science forward at all.

I think it's useful to think of studies like this as a MVP. They're relatively cheap and fast. If they give a null result, you've failed fast. If they give a non-null, then you can invest in further study and iterate.

As opposed to a MVP, such study can give you any results you want.

They provide dirty data but cheaply which can be used to prod a more rigorous study into life. So they do advance science. It isn't all sterile test tubes and lab coats, at the coal face it gets mucky, it has to.

It's the old qualitative vs quantitative debate.

Interviewing 45 people can provide much more meaningful information than sending a survey to 1000 people. I'm not saying it's the case there, I'm not an expert in this domain. But 45 semi structured phones interviews is perfectly fine for a lot of research.

I made 45 consecutive structured calls in my life, many times actually. I always could interpret the result in many ways, most of the time complete opposites were the leading interpretations. Their actual behavior afterwards told me the reality. People on the phone will say a lot of things, and it's very easy to accidentally make them reply what you want to hear even though they don't mean it.

Reply on news.ycombinator.com