My Philosophy On Alerting (2014) | Hacker News Reader

My Philosophy On Alerting (2014)

linuxczar.net

33 pointspmoriarty5 comments

“...over-monitoring is a harder problem to solve than under-monitoring.”

Amen! Too many times I’ve seen the assumption that if we’re logging something then we need to monitor it, and if we’re monitoring then we need to alert on it. And the end result is an operations alert email address getting 30 or 40 emails an hour that serve little more than at assure the systems are still running and increasing the chances that important alerts will get buried and ignore or not noticed at all.

Discussed at the time: https://news.ycombinator.com/item?id=8450147

Interesting how so little of this can be applied to security alerts. E.g.: most security alerts can't be actioned.

A most interesting statement. If security is reported broken or compromised, it's very actionable, surely?

No, security alerts tell you something might be wrong. You have to then perform some analysis tasks and determine if it is a true or false positive alert. Even if it is a true positive you may not have enough info to action,you might continue monitoring or dismiss the threat. You might have to determine the cost of further investigstion and remediation vs security risk.

For example in a past life I was in a position where malware traffic will be detected from some locations but the infected device can't be located. Or some overt webapp scanning is happening,I want to know about it and make sure it isn't excessive but it's not worth the effort of blocking.

You don't really action security alerts and events,you action security incidents(which are security events/alerts confirmed to be true positive). With infrastructure monitoring if you see an alert that means something is wrong,the alert is an incident. Been on the responding side of both types of alerts.

Incident response is very interesting,most plans hardly survive the battlefield.

Reply on news.ycombinator.com