Hacker News Reader

I feel it's the same with the whole kerberos / "cattle not pets" cloud mindset.

Usually when a service conks out, operations will just kill it and roll out another instance. Nice, but you have no way of finding out what actually happened then.

Sometimes this is caused by someone looking for exploits (and actually finding one), or other stuff you'd really rather know about.

View on HN

You mean kubernetes and not kerberos?

Oh yes kubernetes sorry!

You need better observability or tooling then.

Operation teams (and automation) have usually a primary mandate of availability above all, not to investigate any possible failure.

You achieve availability by redundancy, not by running around with shotgun murdering your cattle.

Definitely have multiple replicas, and recreate misbehaving ones, saving logs and data for later analysis.

If you can't have that (and budget allows) keep unhealthy replicas alive and pull them off the load balancers.

In my experience option one works best, and makes option two redundant. Unknown unknowns are a thing, but that's true even when the unhealthy replicas are kept around.

Reply on news.ycombinator.com