Just last month I asked someone on the team to add some alerts. We like everything to be declaratively defined in the repo and we push metrics, and we have an alerting system. We just needed some plumbing to be able to define alerts that get created or updated on every deployment. I said to start lets add an alert for X metric, then we can take it from there.
He took the instructions, plopped them into an agent, generated ~400 lines of change that add the alert. The problem was the entire 400 lines were custom for that X metric. Meaning if we needed to add a second alert, it’s another 400 lines. The entire thing is generic plumbing and defining resources. Any reasonable engineer would make it so that the second alert to add would be 2 or 3 line change to just define the metric and its alerting threshold (and whatever other parameters for the alert) not re-do the entire setup every time for every metric. I felt stupid having to explain that to another senior engineer.