Comment by matsemann | Hacker News Reader

Comment by matsemann | Hacker News Reader

Lots of monitoring setups are either too simple or too advanced. We had a system accepting a certain kind of applications. We would normally get about ~1500 a day, following a pattern with a steady amount during working hours, many in the evening, and almost nothing during the night. Any human could take a glance at the graph and quickly tell if something was wrong or everything ok.

We wanted to catch if something went wrong and people couldn't submit applications. We set up an alarm that triggered "if less than X applications last hour", but that was hard to tweak so it didn't go off during the night and was unusable. The tool had no way to set that rule to only apply for certain time periods or any way of making more advanced rules. But even if it had, I think it would have been a game of whack-a-mole with false positives and obvious errors slipping through.

Instead we could set up some kind of AI "anomaly detection". That was almost even worse. Firstly, because no one could tell us how it really worked, how does it know what we consider an anomaly? I mean, if there is a holiday in the middle of the week and we get fewer applications than normal that's an anomaly, but nothing to sound an alarm for.

Actually, good anomaly detection can take into account what "normal" is for your use-case.

For example, Facebook Prophet (a forecasting procedure, which can in turn be used for anomaly detection) has extensive support for seasonalities and holidays, and would probably work well for the use-case you mentioned.

The pattern is simple. Collect metrics, filter, apply rules, alert.

The problem is, people try to put all of those pieces into a single product. This sometimes works, other times not. "Filter" and "apply rules" is the real work, and it's rather intractable. Some products embed a DSL to attempt to empower users to create these bits themselves.

Presumably the "AI" could write the filters and rules by "training" on some data. The AI can't automatically deduce the dimensions of your data like 'consider the time and day of this request' without instruction from the human. It can't conceptualize 'is a holiday/is not a holiday' without input from a human.

At the end of the day, just collect the metrics and write a small application with whatever alerting rules you want. This might not scale, but it might not need to.

> Firstly, because no one could tell us how it really worked, how does it know what we consider an anomaly?

Some old-fashioned techniques (eg. just triggering on high deviations with adjustment for seasonality) are at least explicable, given a little time. But they don't get much buzz.

My own view is that the untreated problem for most attempts to apply time series analysis / anomaly detection / concept drift / process control is that the time series observed are highly variable to begin with. A highly variable series means (1) you need more data to build the predictions, (2) those predictions are less certain and (3) the false positives will be higher.

Put another way: the need for such tools is a symptom, and a treatment, but not a cure.

Making software more predictable is more value-adding than building ever more elaborate predictors.

I have similar thoughts here. It’s almost like we don’t need to AI we just need the I. What you have described is a fairly straightforward pattern that seems like it could be expressed by looking at the same period (ago) to see if it’s acting roughly right.

It reminds me of a chat I had with a customer in their slack. They wanted build and deploy notifications but only if a build didn’t happen on a day at an expected time. They don’t want 700 notifications a day, it creates a deafness to the signal. The request was comically difficult.

Firstly, because no one could tell us how it really worked, how does it know what we consider an anomaly

This hits the nail on the head for me. Unless the AI magic handwaving is 100% infallible a human in the loop needs to verify whether there is in fact any problem. Traceability is at least as important as accuracy and statistical methods have a very good track record there.

I don't think of anomaly detection as being the end, it's triage. It shouldn't do more than "I found something that doesn't follow the normal patterns, let me page someone to take a look." And you need to use it strategically, it's not for use everywhere. Like anything, it's a tool and it really depends on how you use it.

I'd like to see anomaly detection as a standard part of (image) classification libraries, as it's incredibly useful to know if a given image is "too far" outside of your trained dataset to be accurately classified.

Reply on news.ycombinator.com