Nice article. My only comment would be that performing anomaly detection on the same data that you use to "train" your detector in the first place, is a classic example of a "data-leak" in data science, and very likely to lead to errors in itself.
Nice article. My only comment would be that performing anomaly detection on the same data that you use to "train" your detector in the first place, is a classic example of a "data-leak" in data science, and very likely to lead to errors in itself.
I agree with your comment in general that using test data in your training set is a clear example of an error that will lead to overfitting and a model that generalizes poorly, but am having a hard time seeing where the author commits this mistake.
Then he proceeds to include 12 in the calculation deriving these bounds. This is not really the way to do it. In fact, if he had excluded the anomalous measurement from the training data, the '5' values would have been excluded as outliers as well, given his criteria for defining the bounds.
I agree that this is a trivial point made on a trivial example though, and that it is more a matter of 'sensible definitions' of what counts as anomalous or as training set in the first place. But it's still worth thinking about explicitly though, so I thought I'd mention it.
But thats the whole point of the very well established concept of standard deviation - to look at a dataset in isolation and analyse it.
This may leave you with some false positives (like any system of this nature would). Of course, you could go the route of actually defining an anomaly yourself and building a more sophisticated model (i.e. one not using z-scores as the measure of anomaly), but that's obviously a different scope.