It sounds bad, but it encourages everyone to write more fault tolerant code. Way moreso then a random bigcorp with an on call team.
It sounds bad, but it encourages everyone to write more fault tolerant code. Way moreso then a random bigcorp with an on call team.
So actually, I really don’t like your idea.
We have the same setup in my org, but we get to define alerts ourselves. All our own alerts are built so that they don't go off if the underlying infra is borked, and only if there's something we can actually do on our level. We are being kept honest because there is a big kerfuffle when an incident is reported by customers first (instead of alerting).
- The only critical alert that can actually page people is if the blackbox test fails. Every 30 seconds, it downloads a test image and if the contents don't match the expectation, an alert is raised (with some delay).
- Warning alerts are mostly for any errors being returned from background tasks, but these are only monitored during business hours.
Doesn’t seem like your case at all.
It was a project that was critical to the company, but that was not very reliable, and broke overnight very often, sometimes 3-4 days per week.
I was the 4th dev on it. Everyone else who worked on it before had burned out and quit. The dev before me couldn’t tough it out another 2 weeks and quit before i joined.
Eventually I burned out after a year and quit too. All this to say, that’s great when it works, but when it goes bad it’s real bad hahaha.
Until whoever dealt it just leaves the team. Then it’s everyone’s problem.
If it happens frequently, the guilty is the process, team lead or whoever runs the things.