New study disavows marshmallow test’s predictive powers
anderson-review.ucla.edu
anderson-review.ucla.edu
Hint: if you're making a strong, large statement—e.g. an emphatic claim about an entire category of things—then it's most likely a shallow comment.
The actual result: measures of self-control either weakly or strongly predict positive life outcomes, depending on the measure and how much adjusting was done, e.g.
> [The study] created a new measure of the time each original preschooler waited before taking a bite (or getting the reward) to adjust for variables such as age, gender and experiment conditions.
This study found that the "marshmallow test"—as a single measure—is no more or less predictive than a basket of other measures of self-control the study tested, or any of those other measures of self-control taken alone.
Despite the misleading article and headline, the study itself seems well-designed (e.g. pre-registered), but the conclusion in the headline is utterly wrong as that is not what the study found: self-control matters, can be measured, and those measures weakly or strongly predict positive life outcomes.
Here's an accurate headline: Self-control still predicts positive life outcomes, Marshmallow Test creator finds.
- Self-regulation composite (preschool & ages 17-37) predicts capital formation at 46.
- Preschool delay of gratification alone does not predict capital formation at 46.
- The composite is more predictive partly because it consists of many items.
- No evidence of more predictive power for self-regulation reported later in life.
Are we looking at the same study? I don't see where "no more or less predictive than a basket" comes from, specifically where "no less predictive" comes from.
My reading of the abstract is that, the study found that a measure based on the "marshmallow test" ("preschool delay of gratification", RND in the article body), is not predictive of the outcomes they measured (11 capital formation outcomes).
It also found that a basket of measures of self-control (collected at various ages, RNSRI/RNCCQ in the article body) is predictive of the outcomes, whether you include the preschool measurement or not.
So from skimming the study without even reading the article, it sounds to me like they found that the preschool measure doesn't predict the outcomes they're measuring by itself, and it doesn't contribute predictive power when it's used as part of an index of self-control measured at a variety of ages.
New Study Disavows Marshmallow Test’s Predictive Powers.
And it does. The marshmellow test as a single measure is referred to as RND. Which is a test which measures gratification wait times and is applied in pre-school. Their hypothesis regarding RND:
hyp2: On its own, RND (measured around age four) will have only a very small correlation with the measures of mid-life capital formation.
And they report a confirmation of this hypothesis.
The other hypothesis refers to RNSRI rank-normalized self-regulatory index - which is 4 different components measured at different ages - each component is RND + 86 other measures! This is reported as having a "modest" impact on outcomes - not "strong" as you say. So your reporting seems the more inaccurate to me.
But this is irrelevant anyway with respect to your claim about the headline, which is only referring to the paper's disavowel of RND.
Further evidence of the headline's accuracy and the paper's disavowel of RND is that they also looked at RNCCQ - which is RNSRI minus the inclusion of the RND test from each of the four components. They found that including RND did not improve RNSRI over RNCCQ in terms of their predictive power.
I've always wondered if this test measures more of the child's willingness to please the researcher, and not so much their capacity for delayed gratification.
The marginal utility of the second marshmallow is lower. Then there is added cost in the time spent waiting. Even if you have to wait in the room regardless, I think it’s fair to say there is a cost to delaying the gratification.
Why one extra? What if I just give you 5% of a marshmallow? 1%?
A coherent argument can be made that the extra costs are not worth it. I don’t particularly like marshmallows, but maybe I’ll eat one. Maybe two marshmallows in the future would be higher utility or whatnot than one marshmallow now, but that isn’t always the case.
What if I hold out for 10 minutes only to finally give in, and then the researcher returns seconds later? Better to fail fast than risk that.
At the moment there seems to be a lot of weight behind the idea that we only have so much willpower to burn in a day. So you might hold out against that tempting box of donuts all day at work, only to break down stop at a bar on the way home for a few pints.
I think on some level I've always felt that avoidance is better than fighting when it comes to willpower. First option - avoid the temptation. Second option - give in. Only if #1 isn't feasible and #2 is a very bad outcome, then go to battle.
There are multiple strategies people can take to willpower and life. Some work better than others in different situations.
There are people who don't mind waiting patiently for that 2nd marshmallow. I'm sure there are situations in life where that mindset is beneficial.
Move fast and break things is good for some goals, slow and steady is good for other goals.
This would also be a trait that leads to success later in life.
These studies could be right for the wrong reasons.
As a side note, I have a four year old and have no idea if he even would eat the first marshmallow. Sometime I imagine he would eat it instantly and then he would talk about it for days to us (“can you believe they had a marshmallow on that plate!”). Other times it would remind him of birthday cake (or something else) and he’d just save the marshmallow in his pocket forever. For example, he still has half his Halloween candy and it’s late February.
https://addictions.psych.ucla.edu/wp-content/uploads/sites/1...
> Conclusions These results provide strong evidence of greater DRD in individuals exhibiting addictive behavior in general and particularly in individuals who meet criteria for an addictive disorder.
I don't remember whether I read this in relation to the original study, or a follow-up. But it sounds more plausible than the original explanation.
The question of why some children can delay gratification is different that what how predictive delayed gratification is of life outcomes.
It can also be true that broken promises are a better prediction of future outcomes.
'do' depends on context
I am capable of driving fast, and weaving in and out of traffic on the freeway. But I choose not to, due to the risk/reward trade-off.
If I were in a different situation (e.g. sick child needing to get to hospital urgently), I might act differently.
Measuring my behaviour in one context may tell you little about what I can and might do in another.
>Measuring my behaviour in one context may tell you little about what I can and might do in another.
It may tell you little, or it may tell you a lot. That is why you run a study and see how predictive it is.
If someone drives fast and reckless today, they might not tomorrow. You can still run a study to see how predictive today's driving is for tomorrow. It won't be 100%, but I bet you would see some correlation. Someone who smoked a cigarette today is more likely to smoke tomorrow than someone that who didn't smoke today. That people can quit and can start doesn't change this
This does not mean that the test is not predictive. It means that the index (a bunch of measurements) contains statistical dependencies. From a practical view, the marshmallow test result depends on many cognitive factors unrelated to self-control. The child must understand the instructions, remember them for the duration of the test, trust the provider, value the second marshmallow, and then make a decision. To be of any value, it should have been tested against a standard cognitive battery, which it almost certainly would have failed to improve upon. Cognitive tests have worked extremely well to predict life outcomes for decades now if not centuries.
A small quote: "But this is not a story of fate – of children’s long-term success being determined by their self-control as four-year-olds. It is a story about how children can change: those who are “low delayers” can in fact learn to be “high delayers,” and gain the life benefits that self-control imparts."
So this is more olds than news, but perhaps it is good to be reminded that we all have room to grow (or shrink) from who we were at age 4. I personally would bet high-delayers can also learn to become low-delayers, and I also would bet there are times in life when you would be better off eating the marshmallow now instead of investing it for another 30 years at 5% because the man in the suit told you to.
What I would enjoy, I think, is taking a monthly test battery and uploading that to a central database with other self-researchers and then looking at that in a historic sense to derive ideas to study. Obviously, since one is post-hoc slicing one will find many spurious correlations but perhaps these correlations will yield interesting areas to search around. Does anyone know of anything like this?
https://www.sciencedirect.com/science/article/pii/S016726811...
How can we conclude anything at all about the general population using a sample of Stanford kids?
https://en.wikipedia.org/wiki/Replication_crisis
>How can we conclude anything at all about the general population using a sample of Stanford kids?
it's a well known problem that is rarely brought up, WEIRD bias. Most social science research participants are college students being bribed with extra credit or gift cards
> People wonder why so many government programs fail
This, though, I'm not so sure about. Do government programs fail at a rate greater than those undertaken by other large organizations, like corporations or non-profits?
The problem is the system incentivizes bad social science.
There is nothing wrong with doing a study with faculty children. It is a great idea to get a bigger study done within a budget and timeframe. They write it up stating as much and their intelligent peers understand the caveats.
Unfortunately, the results are so exciting that it starts getting cited and taught by people who leave off the caveats. This is why it is important for someone to go back and follow up on the original experiment with new studies.
The broader implications with respect to the 'integrity' of the scientific work in various disciplines here is almost the whole story.
That we can't count on 'the marshmellow' test is marginal relative to how these issues add up and how we don't seem to have a strategy or set of tactics to address the reality of the 'fuzziness' of science, especially in an era where we need to convince people that 'vaccines are safe'.
I don't agree with dang's thesis that the generalisation is 'shallow'. I think the generalisation is the story here. It's the new, perennial issue. We are witnessing a new phase in the evolution of Enlightenment values particularly because we are coming to terms with the reality of a lot of what is being published these days, it's the heart of the matter. We need much more dialogue about this, especially institutionally.
https://addictions.psych.ucla.edu/wp-content/uploads/sites/1...
> Conclusions These results provide strong evidence of greater DRD in individuals exhibiting addictive behavior in general and particularly in individuals who meet criteria for an addictive disorder.
They don’t draw conclusions about causative success, just a correlation with addiction.
> Controlling for differences such as household income and cognitive abilities ...
So it's a (predictive) IQ test.
Perhaps it disavows the prior assumed basis of "deferred gratification", but not the predictive power of the test.
I predict that in twenty years, no matter how thoroughly this is debunked, I'll still see this treated as true constantly, and, even when what's under discussion is taking action based on its being true, I'll only get eye-rolls and head-pats and plain disapproval/loss-of-face for bringing up that it's questionable at best, then everyone will go on treating it as true.
"Behavioral phrenology" maybe?
But maybe in the days of phrenology, people thought a hooked nose* had a plausible connection to personality as well?
Weird to think about.
*Sorry, this is physiognomy not phrenology. The same basic point stands though.
If our society needs 1% of the population to be accountants (to pick an example) but only 0.1% of the population either have incredible focus abilities or don't find accounting brain-meltingly dull, then at least 90% of accountants are going to feel like they have a lot of trouble focusing at work. Once enough start medicating (legally or otherwise) it's gonna feel to others like they really do have a condition that most don't, but they both kinda do (in a practical sense, they do need to focus better to keep up with their peers) and kinda don't (in that it's sort of our society that's sick, not them—they're just acting like most people would, in that situation).
The marshmallow test was so appealing because it was incredibly easy to perform and seemed like it was pretty close to a measure of the kind of self-control and discipline that's needed to succeed in a variety of life's most important challenges.
Phrenology is bumps on the head right? I think hook nose would be physiognomy. But yes, the idea was that your behavior was due to your brain and your brain was composed of many different parts that each controlled different propensities or abilities. Then it was just a matter of identifying where those propensities lived, in relation to the head and then you could feel for the differences from person to person across the surface of the head. From the naive viewpoint it is plausible, oh, you say the back right section of my head, above the ear is a bit larger so the self-control portion of my brain is well developed? I've always thought so!
Setting aside the marshmallow test, you can easily see how scientific theories about this sort of thing, both right and wrong, easily integrate.
Nutrition studies are more like phrenology in that they are so hard to do with so many confounding factors that you can’t really trust any macro conclusion.
Lots of conclusions from nutrition studies (especially meta analyses) are robust and useful to follow.
Consider the NutritionFacts website as a good starting point: a non-profit which has no ads, no industry "partners", etc - focused on distilling well-designed studies to see what everyday people can use from them.
Even though nutrition is one of the very oldest, and perhaps the very oldest, fields of human study, it still remains in the “butterfly collecting” phase of development as a science. It’s very very hard. I’m glad some people try.
Sounds more like the test is just testing for an adaptation that happens to be well suited to living in a upper-middle class to wealthy environment. If resources are scarce, the kid that takes what they can get now rather than trusting other people will do better in the long run.
Breaking the feedback loop usually involves doing something farsighted, risky, and irrational - for example, risking getting fired from your retail job by studying programming and applying to software engineering jobs in your downtime, or quitting your stable corporate job to found a startup.