Ask HN: Sysadmin uptime gotchas?

Would anyone like to share any semi-rare sysadmin problems they had that caused problems for a website? I had two problems recently that I didn't anticipate which caused some downtime. It's for a website with 700 req/sec, LAMP setup, 1 web server, 1 DB server on EC2.

* Sporadic mysql_connect errors at high traffic periods. Cause was some combination of hitting the limit on max open files and ip local port range on the web server (http://www.tigase.org/content/linux-settings-high-load-systems)

* All database connections from the web server were refused because the database hit the max_connect_errors setting in mysql, which is only 10. Flushed hosts and increased the setting a bit to fix.

Does anyone else have other "black swan" events that they could share to help us soon-to-be-great sysadmins? We don't have to learn everything the hard way :)