Comment by bluenose69 | Hacker News Reader

Comment by bluenose69 | Hacker News Reader

I use it mainly as a data-exploration method. For example, if the Winsorized mean gives a value that differs a lot from the conventional mean, then I might examine the outliers in a bit more detail with tools like a boxplot or a histogram.

The source of the data matters a lot in what methods make sense. For example, hand-entered numbers might involve transposed digits, or missing signs, or decimal points in the wrong place. Numbers deriving from some electronic measurements might have problems with numbers "pegging out" at some limit. In other cases, those numbers might "wrap around". Data that have been examined at an earlier stage might have numbers changed to something that is obviously wrong, like a temperature of -999.999 or something. The list goes on.

My point is that exploring outliers is often quite productive, and comparing means to Winsorized means can be a very quick way to see if outliers are an issue. This is not so much an issue for interactive work, for which plotting data is usually an early step, but it can come in handy during a preliminary stage of processing large datasets non-interactively. It can also be handy as part of a quality-control pipeline in a data stream.

when you talk about Winsorizing, you're talking about applying some subtle sophistication to statistically sparse data... so then you mention histograms, which also apply subtle sophistication, so you best be talking about histograms with unequal bar widths. Equal bar widths is a bar chart. It doesn't get to be called a histogram (pace wikipedia) unless every bar reflects the same proportion of the population.

I'm a confused. Are you saying that a chart isn't a histogram unless each bar represents the same number of items? I'm not sure I've even seen a chart like that before, I'd love to see an example if you've got one handy.

It's what I was taught in statistics class at MIT. But you're right, I rarely see them, people just call bar charts "histograms". What it does is smooth out outliers. There is always (i mean, within reason) a good sample at the mean, but as you go out from the mean you get more and more variance (it's sort of the definition of variance)

A gaussian bell curve is a "population density function" and this definition of histogram reflects keeping the population density meaningful in terms of height. I will try to look for some. But the wikipedia article on histograms does reference them, that's what they are talking about wrt "varying width" bar charts. It's just that there is a rule to varying the width, "x times y is equal for all the rectangles" (oh yeah, you even get to "eyeball it" to make them. Unless you have a massive number of sample points, if you can choose a width parameter that makes your curve look normal, your curve probably is normal, or at least, you can't say that it's not, no matter what the bar chart looks like.

to put it another way, if you follow this rule for a perfect normal distribution, you'll still get a plot of a perfect normal distribution. But if you have a sparse sample that's a jaggy bell curve, it will move the curve toward a smoother curve, accurate in the sense of "this curve is statistically equivalent to your data, so if it looks "normal", probably your data is normal, and you won't imagine a mode where there isn't one.

The parent is more confused than you are, I think. I have almost never seen an histogram with bins of unequal sizes. The claims were dubious so I checked Wikipedia, which mentions this:

> The bins (intervals) must be adjacent and are often (but not required to be) of equal size.

And this:

> A histogram is used for continuous data, where the bins represent ranges of data, while a bar chart is a plot of categorical variables.

To be fair, there probably are variations in nomenclature across fields. But saying that a histogram with bins of equal width is a bar chart is wrong, at least in the general sense.

If every bar reflects the same proportion of the population then that would mean every bar of the histogram would have exactly the same height, because the y axis on a histogram is density or proportion.

I don't get how you can be so confident about a concept you only vaguely remember from a class you took long ago, and that even after reading the relevant article on wikipedia you conclude that therefore wikipedia must be wrong rather than thinking "hmm, wonder whether I'm misremembering things here..."

Reply on news.ycombinator.com