What features are you looking for and how do you rank them?
What features are you looking for and how do you rank them?
Prometheus is by far the cheapest, both for infrastructure spend and for human cost. The only time we spend really working on Prometheus is configuring our service discovery, which is the shipping component you’d have with any monitoring tool. I estimate we spend about 1 developer day a month un Prometheus upkeep.
Logging is much more fiddly, and required a large up front investment. Now it’s running though, we get a lot of value from our setup. It costs much more though, think about $10k/month in infrastructure spend and about 3 developer days a month to maintain.
Grafana is effortless to run, really dead simple. You can consider this to be ‘free’, both for infra and maintenance.
Hope that gives you a sense of this? I think we come out equivalent to managed services for cost when you account for infra and human time, but have far more flexibility in how we use the tools and develop the skills to properly leverage each product along the way.
My company uses ELK, and I personally really dislike it. It's okay when it works — filtering logs with queries is the primary use case, and it's decent at it when it's not returning 503s. But it's also based on Elasticsearch, inheriting all its warts. I really wish someone would build a better competitor. Loki/Grafana isn't anywhere close yet.
One of my pet peeves with ELK is how the indexer assumes all log entries have the same schema. So if one app has the "error" field as a string, and other logs it as an object, then ELK will reject the second one. It boggles my mind that someone thought this was a good technical design. It could easily suffix internal field names with its schema (type, analyzer, etc.) and then unify the fields at the UI level. But no, instead it discards log data.
Sluggish performance (ELK requires enormous machine researches for no particular reason, though the JVM is a major driver) and lack of support for tailing are two other pet peeves. (I have many more.)
That can be fixed by:
- logging to different indexes
- preprocessing your logs so the keys have their own schema prefix, as you mention
The way we've tackled this is to have an official company-wide logging schema. It's just a GitHub repo at gocardless/logging that has an exhaustive list of logging keys, with an explanation of what they should contain.
This has the benefit of encouraging consistent logging practices over many teams, as well as improving the chance your logs will get indexed correctly. If the field type doesn't match, we won't index that field but it will appear in the _log field, where you can do a full text search as a fallback if you really needed to find your log.
It's not perfect though, and I still hate the dynamic type assignment.
My initial draft of this stated that we are using Prometheus/Grafana + ELK across the company, although its still being standardised and turned from individuals creating their own deployments to a proper managed, strategic system with SLAs/documentation.
We're looking at the feature lists of SaaS offerings like Datadog, Logz.io, but I think we're too big to get a sensible price. That said, I don't think we're big enough to justify something like Thanos that turns Prometheus from a really simple application, into a big time investment.
Out of interest, are you storing 2TB of Prometheus data in a single instance? Or spread across multiple?
The 2TB of data is what we have stored across all those Prometheus instances. We use thanos as the entry point that Grafana speaks to, so you get aggregated results.
Thanos as a querier is very simple to setup, and is very low maintenance. We have intended to configure long term storage using the GCS backend for years now, but sadly this project always ended up losing to other (genuinely!) more impactful work.
We hope to do this within the next 6 months though, and reckon the project will take about 2 weeks of our teams time.
For the monitoring angle then, I can recommend Prometheus and thanos as very easy systems to configure. Even for a small team with no prior experience, you'll probably have a good time.
The one to watch out for is Elasticsearch, as that is a fundamentally more complex system in which you plan to store much more data. Loki looks much easier to setup and benefits from the Grafana ecosystem integration, if you're looking for a shorter/cheaper/less featureful option.
You comment on standardising the logging schema is a great idea as well. I'll circulate that around.
Thanks for your help!
I think ELK is the most challenging bit as you outlined yourself but is still very doable.
It's really nice to have some metrics when for instance a service goes down. It's super easy to spot a OOM situation or other vertical scaling issues.
I'm now wondering where the line is between necessary monitoring and bikeshedding? I could look at introducing distributed tracing, but will it actually add any meaningful value?
- Service discovery can get tricky. In our case we manage the physical machines by ourselves so at the beginning we just had to write down static configs, although by now we have automatic discovery (took a few days of development).
- Depending on what you want to monitor, you might need to write your own exporters to Prometheus. However, mtail [1] has been really useful to create metrics from logs without too much work. In any case, you'll have to put time in deploying and configuring those exporters.
- Dashboards and alerts. There are dashboards for a lot of exporters, and there are collections of alerts too [2], but you will need to put time and effort in modifying/creating dashboards and writing down alerts. However, it's a productive effort because it helps in having a better understanding of which metrics are important and how do they relate to the workings of the software you use. Also, PromQL is a pretty nice query language for the purposes of Prometheus.
- Notification integrations. In my case we had to put some time to properly configure a Microsoft Teams integration and a deadman switch channel, but in most cases it will be pretty straightforward.
All in all, you'll need to invest some time in the integrations, but those are things that you need to do in any case. Prometheus itself is pretty easy to set up and maintain, and doesn't stand in your way. No tweaks, no undocumented settings, no bugs. I'm pretty happy in that regard, once you get it running you don't have to worry about it. Storage usage is pretty low even with a high amount of exporters and metrics per node, maybe around 10GB for 60 days of data of a single node? (I'm not sure because Prometheus does some compression and it's not exactly linear with the time or the number of nodes).
And that relatively low investment pays off quickly. The machines we manage use various tools and programs to deal with quite a lot of data at high bandwidths, so performance problems and bugs can be difficult to debug. The Prometheus + Grafana setup has made several times easier the debugging of issues and performance problems, the alerting system helps us prevent outages and we have even discovered issues that were unknown to us. For me, the moment you manage machines with even just a little bit of complexity in the software or setup, it's already worth it to look into monitoring.
1: https://github.com/google/mtail 2: https://awesome-prometheus-alerts.grep.to/