Ask HN: Basic Error Alerting Practices

Hi HN,

Our team currently has an issue with handling and alerting on errors that occur in our application.

Right now, an error that should never happen, ie a bug in the system/unexpected behavior will be logged as an error. Also any connection failures or other transient errors will be logged as an error. These logs feed into the same logging system, and alerts (pagerduty) are generated when any error is recorded.

We would like to set a tolerable threshold for transient errors like connection failures before an alert is triggered, but also be alerted immediately for the unexpected behavior type logs.

I'm sure this is a common scenario for any company as it scales, I'm curious what practices or strategies are used for this.