Simple Alerting for the ELK Stack | Hacker News Reader

Simple Alerting for the ELK Stack

github.com

53 pointsbe_erik26 comments

We're using this. We're not running the full ELK stack; rather, we've replaced Logstash with fluentd. Our devs have two choices for acting on their logs: they can use elastalert (running in Kubernetes) to alert on events that are recorded in Elasticsearch. We've provided a sample elastalert template that can be easily customized to a developers needs.

They can also deploy a custom fluentd parser/transformer in Kubernetes. To make this work, they apply a special label in their Kubernetes replication controller that specifies the name of the custom fluentd parser service. The primary fluentd service pulls the logs from Docker and when it detects this label for a particular log entry, it routes that entry to the custom parser service. This allows us to have a standard log pipeline that works out-of-the-box for most projects but also self-serve custom parsing for the apps that need it.

Stack Exchange's monitoring system, Bosun (https://bosun.org/), also has support for alerting on the results on Elasticsearch queries.

It sounds like they're using a search engine as a substitute for a proper CEP/ESP like Apache Flink. Was the latter considered? I'm rather curious as to why they chose a search engine instead.

ELK is a very standard stack for collecting and disseminating logs. It's a fairly natural, and i think sensible, step to build a pattern-based alerting tool as a bag on the side of ELK.

If you were going to do it more properly, the first choice would probably be Riemann, which has already seen some adoption in this role. There are already plugins on both sides to forward events to the other:

https://www.elastic.co/guide/en/logstash/current/plugins-out...

http://riemann.io/api/riemann.logstash.html

With all respect to the very smart author of Riemann, as a practical matter, its use of Clojure is simply not a hurdle most SRE types are going to overcome to use it.

ElasticSearch is a great search engine. It's not a CEP or SEP. I don't mean to discourage its use, but for this purpose it's not a good fit. It does make sense to tee incoming logs to both as they suit different purposes, but I'd delegate pattern analysis to the latter.

SEP? I know CEP. Riemann is great but I think you are right, there's a lot of people who aren't willing to pick up some clojure for whatever reason.

Using Clojure, especially to slap together Riemann configs, is really not an insurmountable obstacle, especially when something as useful as Riemann is on the line. The more the self-defeating attitude that "Clojure's too hard" propagates the less likely people will be to even try. Most SRE's are pretty smart, I believe in them.

There are some great commercial options in this space like SignalFX and Sumo Logic. The cost of Riemann's learning and implementation challenges would have to be made significantly less for entire teams to make it a viable alternative to those, in my experience.

If anyone runs a team of 25 or more SREs and effectively uses Riemann (and there's more than 3 domain experts), and is not otherwise a Clojure shop, I'd love to hear from you.

> as a practical matter, its use of Clojure is simply not a hurdle most SRE types are going to overcome to use it

And yet, it's the most widely used option in this role. Crazy days.

Can you site something specific for this claim? Are you talking about clojure or riemann or both?

Is anyone running this in production and care to share their experiences? This seems to be a pretty good replacement for Splunk's alerting mechanisms in the ELK stack, which has always been one of the parts I miss the most.

I have been running this. It works well. I also wrote some custom code to send the stack-traces, if we get any, to a slack channel.

That sounds fantastic. Do you have an example of how you extracted the trace? Was it simple to extend ElastAlert or did you have to fork it to add that kind of option?

The trace is basically extracted using Logstash's multiline filter. It adds a tag "stacktrace" to the log entry.

I just made a few changes in SlackAlerter class to customize for my use-case. I'd try to clean-up and send a PR soon.

FWIW the company behind Elasticsearch is happy to sell you a solution for that: https://www.elastic.co/products/watcher

I am using it right now in production. It's worst trait is that it is not clustered and does not share state about what alerts have been sent already. This means you can have no more than one agent running without having duplication of alerts. This also complicates high-availability for obvious reasons.

(Correction/Public Shaming: Its)

My biggest issue with this was that it is purely event driven. So what if your trying to alert on no event? They had no answer for that

Right. This is something you can get quite easily using Riemann.

It's a nice product, just be aware that it is purely event driven. So if you are trying to alert on no event then things get messy

Unless I'm misunderstanding your comment, isn't that a flatline type of alert, which is supported by Elastalert?

No that is just if a value drops off, but that rule will never fire unless an event occurs in elasticsearch matching that query. I found this out the hard way

How does this compare with the log stash alerting plugin that's available? Does it work without log stash (i.e., just elastic search)?

It works against elasticsearch and is somewhat agnostic as to how those documents got there in the first place. I've been playing with it as part of an EFK (fluentd instead of logstash) install.

Do you have a link to the logstash alerting plugin? I'm not familiar with it and I'm interested in alert tools for the elk (and beats) stack.

Are you referring to watcher? That's a paid product I believe no?

Reply on news.ycombinator.com