Hacker News Reader

> One thing that I've observed is that the more errors there are, the less likely people will look at them

It can be even worse than that. At one point I worked on an app that generated north of 500 lines in the PHP error logs per minute per server. In the course of implementing a feature I cleaned up a few of the low hanging warnings near the code I was modifying. This resulted in about a 3 percent reduction in the log file velocity.

When we deployed to prod, everyone panicked, because log file velocity staying about the same was one of the primary things they alerted on, as a proxy signal for server health. That was the one and only time in my career I was reprimanded for cleaning up warnings.

> log file velocity staying about the same was one of the primary things they alerted on, as a proxy signal for server health

I don't want to bash on PHP like it's 2002, but ... that's probably one of the most PHP things I ever heard.

I worked somewhere where people were against raising error_reporting because someone was in the habit of writing "flase" instead of "false" – which worked because PHP – and it wasn't deemed worth fixing :-/

Old PHP days were wild.

'log file velocity' ... that's one way to go about it!

> One thing that I've observed is that the more errors there are, the less likely people will look at them – much less fix the underlying source – so the problem only grows. If you have 1 error/month it's "zomg! An error! PANIC!" but if you have 10,000 every day it seems an insurmountable task to fix it all, "so whatever". I don't think it's apathy as the post suggests, it's just that it can seem like an insurmountable task and whatever you do will just be a drop in the ocean.

If I may broaden this, I think the same principle applies for:

- poor performance

- tech debt/otherwise hard to manage extant functionality

- gaps in documentation

And probably a lot more. If it’s one or a few narrowly identifiable problems, it can be easy to escalate and rally and solve. If it feels pervasive, even trying to address one thing feels insurmountable. And speaking to this from a perspective of being highly motivated to tackle problems like those I mention, and having some fairly significant successes in that regard over recent months, it still feels daunting. But it feels a lot more reasonably possible and more importantly demonstrably valuable the more I chip away at it. But it has taken actual years to get even that far.

> One thing that I've observed is that the more errors there are, the less likely people will look at them

And then you have nodejs warnings

A good logging tool will group similar errors together so one error occurring 10,000 times shows once in the list.

But what I’ve found is a whole lot of errors are things timing out which are kind of an issue, but not something you can really investigate or solve.

Grouping helps, but often errors are distinct enough that they will get duplicated regardless, especially after updates which changes the stack trace. To be honest it's been a few years since I used these tools, but last time it was helpful but very much less-than-perfect.

> a whole lot of errors are things timing out which are kind of an issue, but not something you can really investigate or solve.

Depends what is timing out; if it's SQL queries then you should probably do something somewhere, if it's HTTP connections then you probably shouldn't log them as an error but as info/debug, keep some aggregate statistic somewhere, or just not do anything with it at all (depending on your scale and what you need).

Most logging tools if used correctly separates static messages from dynamic values. Which makes grouping trivial.

Or switch to gRPC :) if it is internal to the data center traffic.

You can still get network errors with that, no? I'm pretty sure I've seen some of those anyway.

The go API has two modes: expose network errors to the caller or “trust Google.”