113 karma · joined January 3, 2026
A few rules that have saved us:
- Persist before responding. Never process inline. Write payload to DB, return 200 fast.
- Idempotency key required. Either provider event ID or hash the payload.
- Async worker processes from queue. Exponential backoff + max attempts.
- Dead letter queue + dashboard. Humans need visibility.
- Alert on backlog growth, not single failures. One failure is noise. A growing retry queue is signal.
- Relying on provider retries alone has bitten us more than once.
That's duplicated work. The developer wrote it, the tester verified it. The reviewer's job is different: make sure the next developer can understand and maintain it.
The framework boils down to three questions: 1. Can I follow this? 2. Can I find this later? 3. Is this where I'd expect it?
Curious how others handle the tension between thoroughness and velocity. We landed on "one business day max to respond" as a forcing function—what's worked for your teams?