Hacker News Reader

Comment by sethammons | Hacker News Reader

Structured logging is my jam. Our unit tests validate error content, errors have all dynamic text in different fields, provide all info to reproduce (including, where appropriate, copy-pastable curl commands to replicate an error to remote api endpoints!) and log aggregators like Splunk can group and aggregate and find relationships.

If I see a spike in errors, I can, in real time, count them by user id, group by region, filter out by some other field, then have the table of events generate a live line chart easily showing the error rates over time, and pin pointing the exact moment an error started.

Less than five to ten minutes after detection or report and I can be talking with another team about a change they released at time $x, including a list of affected users for product teams to reach out to if needed.

View on HN

This sounds awesome - pretty disk-heavy but worth it. How long do you retain these logs? What sort of user/request volume do you deal with?

Do you have any resources/blog posts/keywords for learning more?

For compressed logs, it doesn’t have to be significant. If you can get structured logs into a TSDB you really only need to retain logs in cold storage or for as long as more detailed views or correlation might be necessary.

SendGrid, so billions of incoming requests that multiplex out to at a minimum 8 other log events, plus all errors, etc. We have to run our own splunk instances. We used to store data for a much longer time but as our scale keeps going up so do the costs. We've had to reduce this to a 7 day lookback for higher volume services. For lower volume (in the millions not billions), 30 days to a year lookback depending.

As for blog posts - I'm not aware of any. I've actually wanted to show off what we have but have never prioritized the blog post.

> As for blog posts - I'm not aware of any. I've actually wanted to show off what we have but have never prioritized the blog post.

Please do, I'm interested in learning as well.

Doesn't the curl replication leak secrets, though?

Depends on what you are doing. We mask auth stuff and some known params like an auth header might log as "authorization: bearer $APIKEY". Some things can't be logged. But as a general rule, it is an immense time saver.

Yeah, structured logging has a tendency to do that as well.

Logs are often a treasure trove for system invaders.

This sounds awesome. Could you share some examples of logs you have?

Reply on news.ycombinator.com