Think like a statistician, without the math | Hacker News Reader

Think like a statistician, without the math

flowingdata.com

31 pointsmorisy14 comments

Hmm,

Statistics is already dangerous enough with the math, I shudder to think of it without the math.

--> The danger just begins with the umpteen media-driven conclusions driven by the correlations that one can find in data while neglecting the point that correlation does not imply causation. The danger continues with the fallacious-but-Nobel-prize-winning efforts of Black and Scholes (see collapsed American economy...). Maybe "thinking like a statistician without the maths" means understanding these problems but I see nothing in the article to suggestion this. The article seems to describe the process as "attend to details, think big picture, live-right, think-right, to-thine-own-self-be-true, etc" (nothing but intellectual banalities).

Fooled by Randomness is not a great book but it might be a much better place to start than this article. Still, I would say, learn the maths, darn it.

I think people are missing the point of this article. The author isn't saying that people without math degrees should go out and try to be statisticians. What he's really saying is that anyone with the right mindset can do statistical analysis. I believe this to be 100% true. If you are in the type of position that looks at a lot of data, you will naturally develop an eye for trends and outliers. Then if you are curious enough, you might draft up some conclusions which you then bring to attention to others. They will say, so how does this compare to the population as a whole? So you go back and look through the numbers. Even if you don't have a background in math, you're lucky that you've got Google. You type in "how to handle outliers" or "how to analyze trends in data". You learn about the basics (ie normalization, standard deviation, means, percentages etc). Then you go back and look at your data again. It's a continual process of discovery and one that you can learn through application.

For the record, I really like FlowingData.com. There have been some awesome posts, particularly the one about creating a U.S County thematic map.

If you are in the type of position that looks at a lot of data, you will naturally develop an eye for trends and outliers.

So what do you base your "eye" on? A lot of good statistics involves debunking the trends that people with an "eye" read into the data without them really being there. The human is a fantastic instrument for seeing patterns - it just has the small problem that it can easily see patterns in entirely random data - especially patterns with complex, non-normal distributions (lot of fractal random distributions are great for seeming like they were produced intentionally but without maths, how would you know that?).

It's true that sometimes when you see a trend, it really is there - sometimes. What differentiates a good statistician from a crank or snake-oil salesman is that they use a rigorous method to sift the trends they see.

Just consider that the "Eliot Wave Theory" is huge, utterly bogus theoretical method based on "seeing patterns" of stock market movement that can't be found by rigorous methods. We don't actually need more of that kind of thing.

> So what do you base your "eye" on?

This article refers to everyday people who might look at "ordinary" data. We're not talking about government climatologists or financial analysts here. We're talking about people in Sales and Marketing who might want to know the demographics of their customers, perhaps by location for example. So in this case, I mean quite literally, use what you see in front of you to identify anomalies. Maybe you notice that the age bracket 18-24 year olds seems significantly higher than the rest of the population.

I don't want to discredit the need for statisticians. I'm doing my PhD research on machine learning techniques. But in most practical scenarios, you don't need to resort to such a rigorous analysis. You can get quite far just using run-of-the-mill techniques (mean, standard deviation, normalization, what have you)

Think like a chef, without the food

Think like a chauffeur, without the car

Think like a pediatrician, without the child

Think like a n00b, without the inexperience

I agree with this a lot. Math and statistics are just as valuable to individuals for the style of thinking and discipline, as they are for any particular result. Just like the scientific method useful in any discipline making decisions based on evidence, even if you never need to drudge up those chemistry or physics equations.

My degree is in mathematics. I'm an analyst so I work with numbers a lot, but I use nearly none of my "math background" in my line of work. I don't deal with limits or topologies or rings or primality or anything. Nonetheless, I still feel that my math background helps me a lot. Any time you're in the real world you have to work with intuition and experience and make a boatload of assumptions, and that's a perfectly reasonable way to make decisions if you're good at it. Math's precise, rigorous reasoning helps you challenge the soundness of your assumptions and your intuition, which is invaluable.

> [...] you have to work with intuition and experience [...]

Yes, and you can also do exploratory math in this fashion. You just have to come back and prove everything eventually, if you want to be sure.

My degree is in math too.

I think the article is bogus and kind of dangerous.

Math is a matter of exact calculation. If the axioms are true, the premises will follow. Statistics seems like math but involves a group of hidden assumptions about "the world". It's crucial to know the difference between the two fields.

Like mathematics, statistics offers an array of machinery producing results. Unlike mathematicians, statistician should be "gate keepers" as well as technicians. Ideally, a statistician should be there to say "just because you can your stock market data into a heat equation, it doesn't mean you should plug your stock market data in a heat equation".

Unfortunately, the article seems to entirely neglect this aspect except in banalities that a non-math person would be entirely incapable of applying ("dig deeper" won't help you if you don't know why normal distributions matter and why not everything is normal).

You know, I studied math in school for the same reason. When I was in middle school taking geometry, I blurted out to my teacher, pft, I will never use this crap anyway. She pulled me aside and said, no, not this specifically, but as you grow it will help you develop your mind and your critical thinking. I took it to heart, and years later I am a math major.

I'm not especially bright. But I majored in math because I've always felt like the benefits of a math major are beyond the obvious. Like you stated, it gives you a framework for problem solving. More importantly, it forces you to reason your way through long solutions. It forces you to approach problems from different angles and also, in my opinion, increases the size of your buffer (?). In other words, you hold a lot of info in your head as you work through problems and that helps too. It stretches your brain.

Thinking critically, like anything, is just another skill. What better way to get at it than using it everyday for 4 years? No, I've never used my math background to balance my checkbook. Hell, even as an analyst now I use my ppt skills more than anything putting decks together for clients. Getting through the math dept at my school was a bitch. I have very little natural talent.

Even then, I wouldn't have it any other way.

Program your support vector machines from an R package! Make your graphs in excel! Have only the most fundamental concept of what a relational database is!

All that, and you still won't know what you're doing to your errors when you take a logarithm.

If you know enough to take the logarithm of something, you will undoubtedly know what its doing to your errors as well.

Just what we need, cargo cult statistics.

This article should really be entitled: "Non math things to remember when doing the math" or "Think like a statistician while forgetting stuff you learned in your first statistics class."

These tips are useful if you are doing math. But, I think anyone doing the math would (should?) know that outliers can be important and not to read their own agendas into things.

Statistics without math is really about asking "how did you test for this? Is that the appropriate way to check for this? What about this explanation? Is the causality correct here?" Its not doing statistics without math (which you can't do) but "questioning* statistics without doing the math (which you can, sometimes do).

One alternate title: "Some things you learn in introductory classes aren't very useful in the real world."

Data, data everywhere http://www.economist.com/specialreports/displayStory.cfm?sto...

Might be a relevant read too.

Reply on news.ycombinator.com