Hacker News Reader

I'm a devops engineer (I guess we're called SREs now too) at Localytics. We're not Google-scale, but our stuff isn't small, and a big focus of what we're doing is making sure we don't have to work 24/7. Automation, automation, automation. Handle failure states automatically and only inform humans after the system has fixed itself or if the system can't fix itself.

It's pretty fun.

View on HN

How do you control the quality of that automation code? Doesn't that add an overhead to your operations?

Good testing. I got hired as a Test Engineer in SRE at Google to write automation code tests. Good coverage from unit tests, good integration tests, and good system tests (many of them using fake systems with scripted failure modes). Then finally, a fair amount of testing on "real" machines as new code gets integrated.

Then make sure you have great monitoring because it will still fail at runtime and you're the poor sap on call who has to deal with it.

Reply on news.ycombinator.com