Comment by jjjensen90 | Hacker News Reader

Comment by jjjensen90 | Hacker News Reader

Parent Full thread

I highly recommend Datadog's logging platform. One important lesson I've learned in my career is never to run your own observability platform if you can afford for someone else (whose entire product is observability) do it for you.

I've used ELK (managed and hosted), Splunk, NewRelic, Loki, and home grown local/cloud file logs and nothing has been as cheap, easy, and powerful as Datadog. They charge per million log events indexed but also allow you to exclude log events by patterns/source/etc and they ingest but ignore those rows (you pay $0.10/gb for those ignored logs).

The 12 factor way to do logging is very easy with Datadog, as you can tell the agent to injest from stdout, containers, or file sources, then the application is agnostic to the log aggregator as the agent will collect and send logs to the platform.

Not only is it cheap and easy to set up, it also gives you the option to take advantage of the other features of Datadog that can be built on your log data. Metrics based on log parsing, alerts, configuration via terraform, etc become possible when you ship your logs to their platform.

I've seen our production apps log 10-20k messages per second without Datadog breaking a sweat but I'm not sure if they have any limits.

> One important lesson I've learned in my career is never to run your own observability platform if you can afford for someone else (whose entire product is observability) do it for you.

Why is that?

1. You have to keep your logging, monitoring, alerting infrastructure up yourself

2. You have to monitor, log, and alert on your that infrastructure yourself with something else

3. You usually have to spend more money, both in hosting and in developer/ops time, on getting something mediocre compared to a provider that exclusively does observability as a product

4. Logging etc are a commodity and you should have a really good reason to build or run something yourself if it is not your core competency

5. Observability is much harder than it sounds and providing a cohesive platform that ties log events to traces to metrics to alerts, and keeping it up and highly available is a hard problem that you shouldn't do if observability isn't your core competency/product

Yep, if you can afford it make it someone else's problem. No team should need to have dedicated OPs people that are responsible for keeping the logging platform running. This is a solved problem and costs the company more in engineering time (paying a person to do the job) than the Saas solution.

E.g. 30K/month isn't much when you compare it to paying a sole engineer 15K/year + cloud costs of hosting your own logging solution + any licenses. There's also a slim chance the home grown logging platform will be as resilient as a Saas product.

I for one am very grateful to DataDog's support team as they've been very helpful in debugging logging problems we've had in the past.

A think a more generic advice would be to focus on what brings value to your customers and makes your product better.

> I thought using loops was cheating, so I programmed my own using samples. I then thought using samples was cheating, so I recorded real drums. I then thought that programming it was cheating, so I learned to play drums for real. I then thought using bought drums was cheating, so I learned to make my own. I then thought using premade skins was cheating, so I killed a goat and skinned it. I then thought that that was cheating too, so I grew my own goat from a baby goat. I also think that is cheating, but I’m not sure where to go from here. I haven’t made any music lately, what with the goat farming and all.

Reply on news.ycombinator.com