assuming monitoring is in place and someone responsible for responding to pages is available, our standard process is to ack (acknowledge the alarm), investigate, attempt to resolve, and, if that takes longer than X (15min) or affects more customers than Y, we declare an incident. Worth noting that we are aggressive on making sure that pages (esp. during off hours) are actionable - it must require a human and it must be affecting customers.

When an incident is declared, a slack room is spun up that auto-includes links to a zoom, links to the incident response process, and pages important people. We have someone other than the responding engineer be the "incident commander." Their job is to make sure we dot Is, cross Ts, follow up on action items, and generally keep the ball moving forward. "Amanda, you were going to pull the db records, how's that going? Does anyone have insights that could help John?" They work with support to get a user facing message published. They send out periodic updates so higher-ups know what is going on with a focus on customer impact. The initial goal is to mitigate impact, then figure out a fix. During the blameless postmortem, we focus on processes that failed and aim to remove human components in failure. We use these to also share processes on how people found information, fixed things, etc. From here, we decide on what is a system improvement in need of priority and design work vs things we can address at a individual or team level. We then have an internal SLO that says we will address the fixes within N sprints and higher ups pay attention.

This can all be manual to start and then you can start integrating with tooling. I enjoyed working with splunk for log analysis, graphs, and alerts, prometheus alerts, pager duty for paging, and jira for tracking betterments attached to incidents. Right before I left my last gig, we started with some saas incident/post-mortem tool and it was pretty decent but nothing to write home about. I've forgotten their name and can't find anything similar via Google.

As for when you should do this? As soon as your team is big enough that you need to write things down, share information with others, and have something worth improving.