Winsorized mean
en.wikipedia.org
en.wikipedia.org
I can understand a line of logic that would give rise to something like the Winsorized mean -- after you look at your data, you see some obvious outliers. It feels dirty to just drop those values (which would lead to the truncated mean) because the information from an implausible value is more likely to be near the extreme than it is to be near the center mass.
What to do with those extreme values?
Here's something I now want to experiment with -- bootstrapping the extreme values. Take note of the original empirical distribution. Then, create a new distribution by removing the top and bottom X% of the observations and replacing them with values drawn i.i.d. from the original empirical distribution. This could lead to some values being replaced with the outliers that we originally wanted to drop. After we do this, record the mean. Then create new sample distributions until we have a distribution of new means. What I am curious about is how the shape if this distribution of means will be impacted depending on that X% value selected at the beginning.
What are some well-known distributions that appear to have outliers? A log-normal distribution maybe?
It does feel like it's a very early 20th century technique. Nowadays we have so many tools which would be less feasible for calculators (the people) and more feasible for software.
Recently, we started using the arcsinh instead of the log as well because the function has nice properties[1]
1. https://worthwhile.typepad.com/worthwhile_canadian_initi/201...
It makes nonlinear relationships linear. Makes the model less sensitive, too. For instance if the data spans several OoM, adding or removing one datapoint in one of those orders can generate a lot of skew before the log-linearization.
It's easy to cast the log back to the original distribution by taking the exponent afterwards.
[1] https://en.m.wikipedia.org/wiki/Generalized_linear_model
Read the Wikipedia article for a more formal explanation.
[1] https://en.m.wikipedia.org/wiki/Variance-stabilizing_transfo...
I don't really like the Winsorized mean but not for the reason you list. I think the main issue is that you are assuming exactly the top and bottom 10% are outliers instead of looking at the actual data distribution to see what the outliers are then using a similar replacement technique on only the outliers.
https://projecteuclid.org/journals/annals-of-statistics/volu...
How does it do on normal and other common distributions?
All of them. Which is why outside of a handful of contexts, the consensus in statistical modeling nowadays seems to be not to worry about outliers unless the values are completely unreasonable or there are a suspicious amount of them.
As to your bootstrapping idea, why not bootstrap the entire distribution, why only the tails? If you only bootstrap the tails but allow draws from the entire empirical distribution then you are changing the underlying distribution.
- A world with a different distribution than the one you are trying to fit
- A measurement environment subject to bad contacts or noise spikes or experimental mistakes
- A reporting system with occasional typos
- etc
Seems to me what to do with the outliers should be informed by some understanding of the environment. And in some case, noted aside while waiting to see if there is more data "out there" in the outliers' vicinity.
In some cases, a replacement for the outlier might be "nearby" while in other cases we know nothing about where the replacement should be.
Like 1002, 998, 1004, 48723, 2104, 1003, 997...
Estimating the deviation from the mean of the last n readings and ignoring ones too far out works well. Also calculate the percentage of bad readings and have a way of displaying it.
Both methods give you better metrics for comparing periodically, like day over day or week over week.
The source of the data matters a lot in what methods make sense. For example, hand-entered numbers might involve transposed digits, or missing signs, or decimal points in the wrong place. Numbers deriving from some electronic measurements might have problems with numbers "pegging out" at some limit. In other cases, those numbers might "wrap around". Data that have been examined at an earlier stage might have numbers changed to something that is obviously wrong, like a temperature of -999.999 or something. The list goes on.
My point is that exploring outliers is often quite productive, and comparing means to Winsorized means can be a very quick way to see if outliers are an issue. This is not so much an issue for interactive work, for which plotting data is usually an early step, but it can come in handy during a preliminary stage of processing large datasets non-interactively. It can also be handy as part of a quality-control pipeline in a data stream.
A gaussian bell curve is a "population density function" and this definition of histogram reflects keeping the population density meaningful in terms of height. I will try to look for some. But the wikipedia article on histograms does reference them, that's what they are talking about wrt "varying width" bar charts. It's just that there is a rule to varying the width, "x times y is equal for all the rectangles" (oh yeah, you even get to "eyeball it" to make them. Unless you have a massive number of sample points, if you can choose a width parameter that makes your curve look normal, your curve probably is normal, or at least, you can't say that it's not, no matter what the bar chart looks like.
to put it another way, if you follow this rule for a perfect normal distribution, you'll still get a plot of a perfect normal distribution. But if you have a sparse sample that's a jaggy bell curve, it will move the curve toward a smoother curve, accurate in the sense of "this curve is statistically equivalent to your data, so if it looks "normal", probably your data is normal, and you won't imagine a mode where there isn't one.
> The bins (intervals) must be adjacent and are often (but not required to be) of equal size.
And this:
> A histogram is used for continuous data, where the bins represent ranges of data, while a bar chart is a plot of categorical variables.
To be fair, there probably are variations in nomenclature across fields. But saying that a histogram with bins of equal width is a bar chart is wrong, at least in the general sense.
I don't get how you can be so confident about a concept you only vaguely remember from a class you took long ago, and that even after reading the relevant article on wikipedia you conclude that therefore wikipedia must be wrong rather than thinking "hmm, wonder whether I'm misremembering things here..."
Rust's `cargo bench` "winzorizes" the benchmark samples before computing summary statistics (incl. the mean).
https://github.com/rust-lang/rust/blob/master/library/test/s...
I tend to benefit the most from seeing the entire distribution visually, and that helps me decide if I'm looking for a median, a "normal" mean, a "mean minus some weird outliers", or something different entirely.
Does anybody happen to know of a good visual guide for how different measures of central tendency apply to various distributions? Anything that emphasizes pathological cases is helpful.
-99, 1, 1, 1, 1, 1, 2, 2, 2, 98, 99
Here, we get:
mean: 9.91
median: 1
Windsorized mean: 18.91
truncated (Olympic) mean: 12.11
2-truncated mean[1]: 1.43
[1] this is when I remove top and bottom 2 values and calculate the mean of all the remaining values.
If you're monitoring response timings on a server, for example, the median might be very close to 0, and it won't shift unless a majority of the distribution slows down. If you take a winsorised mean, you can trim useless long response times that mess with the mean, but still see if e.g. 1/3 of your responses are suddenly slower than normal.
The windsorized mean reduces the weight of outliers, so it can measure "how many outliers are there?" instead of "how extreme are the outliers?"
However some frames may contain unwanted outliers, for example if a satellite briefly passes overhead it will appear as a very bright streak in only one frame.
By winsorizing, outlying pixel values can be eliminated while still maintaining the same number of samples per pixel as the rest of the stacked image.