In this case, a choice is more insightful than A/B.
What I find more interesting is that people give a shit about shower heads.
> We created a secret landing page. The product and design were identical. The only difference? One was labeled “Made in Asia” and priced at $129. The other, “Made in the USA,” at $239.
It's odd that they changed the text AND nearly doubled the price. They seem to attribute the conversion rate dropping to the text change, though.
And they also say: "The visitors were given the choice to either buy the Made in USA or the Made in Asia version."
A/B would be randomly showing either ONLY the USA with higher price to some people and ONLY the cheaper Asian one to others.
However, even that isn't apples to apples as the price is obviously different.
Seems more likely they don't understand A/B.
The generally accepted answer is that the only way to know for sure that people will buy something is to get them to pay you money or get them to commit to pay you money.
This falls broadly under the concept of the "lean startup" where you focus first on proving you can sell what you intend to build, and only build it after you have cash commitments from customers to buy what you intend to build.
The logic behind lean startup is that it's far more likely your startup will fail because no one will pay (or pay enough) for what you want to build than it is likely to fail because you can't actually build the thing you want to build. The later case is of course possible, but in practice far more startups fail because of lack of sales than from lack of technology.
Economists deal with this a lot, and being economists they create models or modify existing models to account for this gap. That certainly seems to work, to an extent, but only when applied to populations rather than individuals.
On a less academic note this is a major problem for science that relies on surveys, because even when anonymous people have an awareness that they're "speaking" to someone and being "judged" in some way. People, even in that moment alone with a survey, want to reinforce the image they have of themselves. When asked by a survey, "What do you want from a new newspaper?" Very few people respond, "Celebrities, scandals, and lewd pictures." People often skew to asking for thoughtful, long-form, in-depth reporting.
BUT... then they buy tabloids and click on bait, and they don't read the complex, nuanced, long-form stuff. If they aren't even consciously aware of that, short of getting them in a behavioral lab, how do you tease that out of them? Well structuring survey questions with redundant questions phrased differently can help, you can get a sense of someone's overall "sentiment" for example, but it's still limited for the reasons described above.
A few interesting examples from :
https://scholar.harvard.edu/files/laibson/files/how_are_pref...
https://pubmed.ncbi.nlm.nih.gov/36434890/
https://www.sciencedirect.com/science/article/abs/pii/S00472...
>BUT... then they buy tabloids and click on bait, and they don't read the complex, nuanced, long-form stuff.
I think this is a solid post overall. That said FWIW, I'll add that I think there can be a further legitimate further layer to this: people can express wishes for how the world would be as part of a desire to change themselves too even while failing to reach it (or reach it entirely) and that's not necessarily bad or wrong. Ie, one of the most effective ways to avoid things like junkfood or alcohol is to make it even mildly more inconvenient by just not having it in your house in the first place. If you're hit by a craving, and whatever it is is right on hand, then odds are vastly higher that you'll indulge said craving whether your higher order processes think that's a good idea or not. It's easily to rationalize, you already spent the money on it. But some care around yourself and further habits (like going shopping after a meal, people tend to buy far less impulse if they're feeling full then very hungry) can help create an environment that is more conducive to making the right choices. I don't think it's purely a matter of willpower.
So in a survey that invokes "how would you like the world to be?" sorts of questions I don't think it can simply be dismissed as "revealed preference" (though that can also be very important!) or hypocrisy if some people wish the world was different even if they deal with the world as it is, or if they wish there were fewer temptations or more powerful/effective tools to deal with them even if they do tend to give into temptations. It's precisely because temptations are tempting that someone might wish to make them more difficult right? Like, I don't think it'd be necessarily wrong or strange if an alcoholic supported higher alcohol taxes or fewer locations or whatever.
Again, not to disagree that sometimes people absolutely are saying things they don't really mean out of concern of judgement or the like, just further agreeing that it's really complex to tease it all out.
Keywords to search for are "implicit" vs. "explicit" preferences
"Sure this alcoholic says he wants to quit booze, but when we put a discount liquor shelf at the checkout where he shops for groceries, he started buying whiskey again. What a hypocrite!"
But if it’s around, yeah, I’ll eat tons of it.
“Revealed preference! Actually you wanted to eat a ton of junk food! You were lying to yourself!” No, I actually don’t want to, in fucking fact.