You can't test everything in production.

Be it an infra change or application. It should always be like:

Staging -> Canary -> Production

If you invest time in making the above pipeline automated it will cause less outages.

P.S we use feature flags but it won't solve many other challenges around infra and application changes.

Some examples: - Changing from Classic ELBs to ALBs (We care about latencies so ALBs served 1% traffic in the start) - Zero downtime single mysql master upgrade. (ProxySql) - New k8s controllers. - New metrics adapters.

Staging should be like a playground to test all edge cases before you go to canary and then to prod.