This sort of methodology sounds sketchy to me - how much can we really learn from this? Does it reproduce? If it does, how do we know there isn't some other cause?
It turns out this paper from Talhelm & Dong supports its methodology by citing a paper by Talhelm et.al. and one by Dong et.al.
This should be the top comment. This thread is chock full of pop sociology, to the degree that I really wasn't sure how to respond to much of it.
That paper compared Chinese people with Chinese people, where both groups had been assigned randomly and forcibly by the communist authorities to become agricultural workers in wheat-cultivating regions or in rice-cultivating regions.
The only confounding factors could be other geographic differences besides their major crops.
The point of the paper was to exploit this unusual historical fact as a social experiment that has eliminated most confounding factors that exist in other comparisons, like the factor mentioned by you.
Same way as always in the non-physical realm: we don't. Luckily, perfection may not be required, adequacy may be adequate.
It still might have flaws, but it's not like they just got people to draw charts and interpreted it as 'collectivist' and 'individualist' for the first time in this study.
They’re just reusing flawed techniques from flawed research.
From my brain's obsessive make-a-fun-analogy circuit:
1. Our research shows we can count umbrellas in aerial photos to predict future rainfall. This Predictive Aggregate Umbral Coverage will revolutionize climatology!
2. Our new research using PAUC [1] shows the country of Elbonia will become a desert in a decade.
> ... Other kinds of errors are more characteristic of poor science. When I was at Cornell [1945–1950], I often talked to the people in the psychology department. One of the students told me she wanted to do an experiment that went something like this—I don’t remember it in detail, but it had been found by others that under certain circumstances, X, rats did something, A. She was curious as to whether, if she changed the circumstances to Y, they would still do, A. So her proposal was to do the experiment under circumstances Y and see if they still did A. I explained to her that it was necessary first to repeat in her laboratory the experiment of the other person—to do it under condition X to see if she could also get result A—and then change to Y and see if A changed. Then she would know that the real difference was the thing she thought she had under control. She was very delighted with this new idea, and went to her professor. And his reply was, no, you cannot do that, because the experiment has already been done and you would be wasting time. This was in about 1935 or so, and it seems to have been the general policy then to not try to repeat psychological experiments, but only to change the conditions and see what happens.
It is not so anymore, psychologists are taught to design experiments showing that changes in an independent variable correlate with a dependent variable. I'm not sure when the change came, but I'd guess that in 1970s it was so already.
I was also surprised to see that this theory has been published in reputable journals (like Science and Nature Communications). So, odds are that critics haven't made strong arguments on this theory so far.