You need better observability or tooling then.
Operation teams (and automation) have usually a primary mandate of availability above all, not to investigate any possible failure.
Operation teams (and automation) have usually a primary mandate of availability above all, not to investigate any possible failure.
If you can't have that (and budget allows) keep unhealthy replicas alive and pull them off the load balancers.
In my experience option one works best, and makes option two redundant. Unknown unknowns are a thing, but that's true even when the unhealthy replicas are kept around.