Does AOL count? I built the mail system. It was kind of a big deal. We pushed 4,000 TPS through servers less powerful than an iPhone. I also spent time at Akamai, though I admittedly worked on a tiny, isolated test lab (150 servers or so, a fraction of a percent of the "real" server base).

I have worked with some amazing people, at all scales, and I have never seen a deployment that managed to completely avoid manual logins. No, you aren't logging into a few hundred hosts manually (well, there was this one shop... but let's not hold that up as exemplary). But you're probably logging into a few to test things out manually before you decide what your automated deployment script will be rolling out.

Even when you do deploy a fix, with a large sysadmin team it's good to know WHO exactly ran that one-off deployment script. When I said "auditing", I wasn't even thinking of PCI, or auditing against malicious actions of any kind - just simple troubleshooting forensics. But we have a great, robust way to see exactly who modified a file and when; it's the file system. Why reinvent it? (I just spent the last day trying to figure out who edited a script. Why, "postgres" did.. of course it did.)

Should you aim to do these manual config changes in the test lab first? Of course. Do you always succeed? No. Some things only get tested in the big lab. Reality intervenes. I try to plan for it.

Mind you, I'm arguing this partly to see if I can be talked out of it. I think I believe it, but I've been wrong about way bigger things. If you've achieved my automation nirvana, and you roll things out to a few thousand servers, never manually, and never wondering who did it, tell me how it works.