Hacker News Reader

Comment by efficax | Hacker News Reader

This is a terrible idea for observability. It would be difficult or impossible with most observabilty platforms to make it parse the http response to determine error responses, but when my systems are sending out a ton of 422s or 400s that's a useful signal to me that something is going on. I'd highly recommend using status codes to indicate application level behavior.

View on HN

I think I agree with the article, because I just got bitten by exactly this. My API was returning 404 as a fairly common response when a user makes a typo, but Twilio's observability code was treating that as a serious error it needed to alert me about.

> My API was returning 404 as a fairly common response when a user makes a typo, but Twilio's observability code was treating that as a serious error it needed to alert me about.

Then Twilio's obesrvability code is broken. Requesting a non-existent resource is not a serious error if resources are requested by hand-constructed URLs.

This is one of those tricky philosophical points on web admin, because there are two sources of 404s:

- random clients on the Internet poking around at stuff, which you have no control over and causes no harm so should not alert

- an error in the links or code of the pages you ship to clients, which will cause a consistent degraded user experience and you should be notified about.

In practice, the best way to solve this I'm aware of is not via observation of server-emitted 404s at all... It's via a back-channel for clients to report errors (possibly authenticated, so randos on the web can't fudge your stats) and then tie alerting to that back channel. So you don't track 404s to /foo, but you do alert on your clients screaming at you that they got a 404 trying to access /foooo.

Of course, this solution requires your end-users to enable JavaScript.

You can monitor for the second case and react accordingly, you just don't monitor for them in the same fashion as 5xx class errors (which yes, despite what this article is about, I'm still using that term).

For 5xx errors you should typically monitor against a baseline of 0.

4xx class errors you want to monitor for substantial deviation from previous averages. This is a good indication if you broke something, vs client behaviour changed. Remember, monitoring isn't always going to "give you the answer", as much as alert you to a possible issue so you can investigate further.

Moreso, in a SaaS application, monitoring for deviations in 4xx rate by client is also helpful. An alert on a single client is likely not a system-level issue, but a devaition across all clients likely is.

Wait - it's exactly the "random clients... poking around" that I'd want the alerts for! 404s occurring due to intentional but broken requests is fine - the caller will know about it and deal with it as necessary. But either way, it is a reasonable argument in favor of not using 404 in the case an endpoint was matched but the specific resource id/path was not. It's not entirely dissimilar to the distinction between "file not found" and "directory not found".

If you alert on 404s to URLs that nobody should expect to exist, you create a vector by which malicious actors can wake up your oncall staff at 3AM. That's probably not something you want to do to your oncall staff.

Depending on what sort of service you're running that might actually be justifiable, but I wasn't assuming a threshold breach for 404s would be waking people up in the middle of the night. At any rate, any such alert is always somewhat vulnerable to that problem, regardless of how the error codes are being triggered. Sounds more hypothetical than likely-in- practice.

Does that mean Twilio sends an alert every time some random webscraper tries to GET some favicon or /admin path that doesn't exist on the server? Doesn't that happen hundreds of times each day?

This is probably the biggest reason I like the author's approach: a lot of tools have assumptions about what, eg, a 404 means that might not match what it means as an application error. For example, my API was also returning quite commonly as my frontend checked the existence of various records. As a result, my chrome console was flooded with red error alerts about failing requests (404s), even though each request had "succeeded" just fine.

In another case, I had a site that used http basic auth. An xhr api request returned an expected 403, which resulted in the browser suddenly concluding the basic auth (which was unrelated to the api call) was invalid and the user needed to be reprompted for credentials again.

Both of these could be argued as browser problems, but that's the point: browsers (and observability tools and many other things) think they know what a given http status means. Using http statuses for app-level errors often breaks that.

> An xhr api request returned an expected 403, which resulted in the browser suddenly concluding the basic auth (which was unrelated to the api call) was invalid and the user needed to be reprompted for credentials again.

That sounds like it was a mistake on the part of the api. The browser should only prompt for credentials if the response headers include a WWW-Authenticate header, but that header should only be included with a 401 response (at least according to MDN).

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/WW...

> It would be difficult or impossible with most observabilty platforms to make it parse the http response

Yea, because they've all been built on top of the same faulty assumption. This isn't a counter argument to the OP in any way and I'm kind of surprised you're not the only one bringing this up as some kind of gotcha.

Obviously platforms would need to be handle and treat http and application layers differently. This is as much a given as the fact you'd have to update all your client code to wrap and unwrap the application layer on top of the http layer.

The observability of your application would actually increase as you'd be able to differentiate between an increase in application failure modes from an increase in http failure modes. That's actually really valuable.

> when my systems are sending out a ton of 422s or 400s that's a useful signal to me that something is going on

Why not ping the monitoring when you encounter the business error?

From that perspective, a stream of HTTP errors could indicate a more specific kind of infrastructure failure vs just clients sending garbage.

Happy to admit I'm wrong though, it's why I posted it here, see what others think

Yikes, no. Other way around in fact!

Observable errors mean that there is an actual error and triggers error handling to the user. By saying, that for example, a search for a user in a list returns no results is a 4XX you are implying that the path is an error when the response is in fact, valid and useful. More than that, you now have cross-wired a REAL 4XX error (for example the API URL was changed and no result will work to it) with this fun pretend 4XX concept that ends up confusing your error handling ('Endpoint cannot be reached with this URL.') with ('Your result has no entries.'). Finally, conceptually speaking, it's not a 404. The URL you requested was handled by the endpoint you intended to reach. That is the exact opposite of a 404 for example.

To me, conceptually it's just not an error in most cases and even when it is, it's likely an error on the other side.

Reply on news.ycombinator.com