Why the Normal Distribution Is Vanishing | Hacker News Reader

Why the Normal Distribution Is Vanishing

alexdanco.com

23 pointsFuturebot24 comments

This looks bullshit to me. He says the normal distribution is vanishing, and to prove himself, he brings up some charts which do not look like a normal distribution. Like the first smartphone-related chart:

"You’re looking to buy a new smartphone. What does your demand distribution look like? Is it normally distributed? Absolutely not. It looks like this:" [ non-Gaussian 'curve' ]

Okay, what is a 'demand distribution'? I am pretty sure the market demand for smartphones does not look like the graph he's showing. Etc. etc.

Pretty sure author is just making stuff up. There's no content to this at all. One has no ground to make such empirical claims without showing any data backing it up.

whole article comes only with 'hunches' and not a single data point researched.

it conveniently ignores many facts and makes up some, like Androids being the cheaper, low quality choice, ignoring there are devices sold for as much as iPhones. one could argue that those are sold in lower volumes, but then that'd be a gaussian distribution

also, of course economics stats are not a good fit for a gaussian. they tend to stabilize around exponential. that's why the price pyramid is a pyramid.

I have to concede that by looking at some data, it seems there is a exponential distribution going on for all price ranges except at the highest end, but that's more a hint of the product price/features space being discontinuous than we being in a post scarcity society where gaussian/exponential distributions suddenly vanish.

but anyway, the fact remains that the data is out there, even for the other field, this on smartphone is but one of the many he didn't research preferring to follow intuition. and human brain is very bad at applying intuition to statistics.

http://image.slidesharecdn.com/tomiahonen-engagewithmobile-w...

http://4.bp.blogspot.com/-uY7kRAl6XfA/URibddfrTqI/AAAAAAAAPL...

>it conveniently ignores many facts and makes up some, like Androids being the cheaper, low quality choice, ignoring there are devices sold for as much as iPhones.

He absolutely does not ignore that fact, in fact he specifically says that the better end of the market is either iOS or high-end Android phones.

Obviously (and all market data point to that), the Android phones that sell more are not the high end units, but the cheapo units that dominate the "free with subscription", "prepaid" etc market.

> This looks bullshit to me

It's obviously bullshit since the guy analyses ZERO data and says the normality is dead. Obligatory facepalm.

And obviously Android smartphones pricing look NOTHING like what he has drawn, there are cheap as well as high end Android devices out there at about every price point.

This is what's commonly -- and pejoratively -- referred to in the sciences as "logic without evidence".

> "logic without evidence"

When I was in school we called these people "philosophy majors." ;-)

I know you're saying this in jest, but a good philosopher will account for empirical knowledge in his arguments, when it exists.

I know it's cool to rip on philosophers, but they're incredibly useful to the sciences: they're usually the first ones to point out circular (or otherwise fallacious) reasoning.

>I am pretty sure the market demand for smartphones does not look like the graph he's showing. Etc. etc.

And I'm pretty sure it does look like that, judging from all the people I know (most either have a crappy whatever smartphone or a high end iOS/Android model, with the middle ground, which is the main bulk of a normal distribution under-represented).

It also matches all information I've seen about the mobile market, user behavior (e.g. how iOS dominates profits and mobile views while still having a much smaller userbase, etc).

The normal distribution is important because it arises naturally when the preconditions of the central limit theorem hold. But you still have to use your brain - you can't unquestioningly assume that any random variable (or sample or whatever) you will stumble upon will be approximately normally distributed.

This is the right answer.

Quite a lot of things are not normally distributed. Often this is because there's more than one generating mechanism. For instance you may have a financial market where normally things go up and down smoothly, people are reacting to news as normal. But then there are also panics, where people are not behaving normally.

There's also cases where the past influences the future, and drawings are not identically and independently distributed (though Wikipedia tells me there's weak variants of the CLT).

The conditions under which the Central Limit Theorem hold are complicated (in particular, weaker and weaker conditions have been shown to be sufficient even fairly recently). But intuitively, they boil down to two conditions that are fairly sensible:

1. The individual observations can't be too extreme. Bounded variance, for example, is sufficient, but weaker conditions suffice.

2. The observations must be somewhat independent. Independence is a strong condition and sufficient, but again weaker conditions suffice.

If 1) doesn't hold, a massive outliers could swamp all other observations. If 2) doesn't hold, for example if one underlying factor influences all observations, all bets are off: the observations could "conspire" to be fairly large (or fairly small), thus giving rise to fat tails inconsistent with the normal distribution.

(Having IID (independent identically distributed) observations, btw, is not sufficient, if the distribution itself is too fat tailed.)

Nice. I was aware that you could weaken independence, but not bounded variance. Without bounded variance, I thought you would get something like Student's t.

The author is making an observation based on market knowledge and long term trends.

Yes, he's not mouth-feeding us with the relevant data points -- we can search for them ourselves and see whether they match his observations or not.

Once we get over this, in the same way that we don't expect any conversation or comment in real life to come with charts and figures, we can discuss and try to verify where his assumptions/deductions are correct and whether they match the data that we have access to.

Being unsubstantiated is a different thing from being wrong.

If anyone else is curious about the second picture, I tineyed ( http://tineye.com/search/863a041e2607e3405d2dcec1fb148bce471... ;-) it and found two agreeing attributions. One from a 2010 piece by Terry Tao:

https://terrytao.wordpress.com/2010/09/14/a-second-draft-of-...

' Fig 23. A "living histogram" of 143 students at the University of Connecticut, with females in white and males in black. (Source: The Hartford Courant, “Reaching New Heights,” November 23, 1996) Compare this non-bell-shaped distribution with Figure 5. '

And another also mentioning U. Connecticut and crediting the photographers in the image itself: http://image.slidesharecdn.com/visualiationofquantitativeinf...

> The world’s shift from Normal, Gaussian distributions of demand towards bifurcated, two-tiered distributions is a natural consequence of our shift from a world governed by scarcity to one governed by abundance.

FWIW, others (e.g. Steve Keen, of "Debunking Economics" fame) would convincingly argue that the world never fit into economist make believe normal distributions to begin with.

This article may look like science, but it's actually just an opinion. Don't be fooled by the graphs.

This article hints at the inequality implications, but I think they're likely to become increasingly important. We're already seeing this with creative works: the vast majority of works earn zero or near-zero, apart from a few massive global hits.

Inequality obviously also affects people's ability and willingness to pay for things. The lower end products are not just cheap but increasingly "free": either ad-surveillance-supported or paid for by the exploitative gambling-like "free to play" systems. Again, it's well known in F2P that most of the revenue comes from a tiny fraction of users.

This could be a high-end news source like the New York Times; it could also be a small, niche site like Stratechery.

He's massaging the examples here. Just because NYT is somewhat above the sewers of the Internet doesn't make it high-end.

There's still a normal distribution with relatively little demand for vile content (shock sites), more for gossip sites, a lot for Buzzfeed and other click bait factories, including what fills many Facebook feeds, then less for NYT and the like, finally very little for niche, truly high quality sources (blogs run by hobbyists, hackers, professionals, academics).

>He's massaging the examples here. Just because NYT is somewhat above the sewers of the Internet doesn't make it high-end.

That's beside the point (not to mention that when it comes to perceived quality, the vast majority of people would call NYT high quality, even if they don't read it).

The point is most will either go to few specific sources or to a few generic aggregators, instead of breaking their surfing across a large distribution of sites (as people did more in mid-nineties).

Data is wrong on examples.

Mobile Phone = "Good Enough Android" and "iOS and Flagship Android"

http://www.gartner.com/newsroom/id/2996817

Apple had 15% of Worldwide phone sales in 2014. Samsung had 24% which is 39% for both and the sales and demand are fairly similar to what people are willing to pay. He had data he just didn't want to use.

Author tries to find some anecdotes that would verify his claims, without actual data of course. I would like to know for example, what is the actual distribution of money people paying for a cellphone (without contract) in different countries? Or the distribution of topics that people are looking in Facebook and elsewhere in Internet. Answer would probably not be a Gaussian but quite different than what author claims.

I was really hoping for something about how marketing identifies consumer market segments and custom tailors product offerings and prices to make the best margin they think they can get. Even so, I'd guess that for commoditized products, there would still be a normalizable distribution of prices offered within the confines of a targeted market segment.

a single agent's preference might not be normal, but the average agent's preference is surely asymptotically normal! (as the number of agents tend towards infinity)

[edit: of course not everything is asymptotically normal. a sequence of Cauchys do not tend towards the normal distribution.]

Reply on news.ycombinator.com