Hacker News Reader

Comment by ath0 | Hacker News Reader

I support these 5 levels of logging, but given the discussion below I also would like to highlight a lesson from an SRE talk I once attended[citation needed] that really there are only three levels of alert:

1. DEBUG - Look at this when debugging, otherwise no one will ever see it. 2. TICKET - Open a ticket and have someone on your team look at this when it gets to the top of your queue. 3. PAGER - Drop what you're doing and handle this, now.

Of course, you want to track INFO and ERROR type messages because a sufficient number of them might cause someone to raise a ticket... but at scale, you probably should've built that monitoring already, and just drop INFO/ERROR down to DEBUG.

View on HN

It's nice to have articles like this to give structure to the conversation, but there's no single solution. Python has default levels of; debug, info, warning, error, and critical, but doesn't really give any guidance to what each means. At one company we tried to create some structure, but 3rd party libraries would flood messages at the "wrong" logging level.

I like these 5-levels and the 3-levels you talk about seem like a paired down version that reenforces what each level means.

To illustrate the "no single solution," we tried to use logging for command line output, neither of those systems really has a level for that. We probably should have created a new level between INFO and WARNING because of chatty libraries.

In practice I've found the DEBUG and TRACE are usually best added to code when troubleshooting. When you're writing the code you don't have a good idea what info you'd need when troubleshooting, so it's overly verbose but still missing what you'd want. When trying to stash all logging either locally or centrally (because adding logging and re-running isn't always an option) it can affect performance.

I think a good log system should give you the ability to store everything but filter quickly. So for example I'll set up my systems to store everything down to LOG_DEBUG because I might want that information. But when I'm actually looking at the logs I like being able to exclude the DEBUG messages and look at the general behavior, then drill down when I think I understand what the sequence of events is. Good log systems should support this without making me filter as I'm storing the data.

Also in UNIX systems I wish there was a way to generically pipe the output of a command to the system log while preserving the return code in case the command exits. In BASH you can configure the pipe behavior to propagate errors to the calling shell, but in normal shells the last return code is the one that gets stored, which is not usually what you want. It would be really cool if the logger utility captured return codes from upstream and returned them so that I can use it in the antiquated SystemV init scripts I have to maintain.

That may work for an application, but there are many errors that occur while using a library that are expected/handled etc by the application. Perhaps separate logs can help.

There are also messages which should not trigger a page themselves but will provide useful context when responding to one.

That's DEBUG

Reply on news.ycombinator.com