I once took down HN by submitting /topcolors. The moment it reached the front page, HN ground to a halt. I quickly fired off an email to sctb who cleared it up within a minute or two (by wiping it off the front page, not by fixing the caching, which is totally valid), otherwise it could have taken much longer to figure out what had happened.

Caching is a hard thing to get right. I try to use “vegeta attack” to load test pages as I make them, but the measurement is the problem. Unless you already have a profiling pipeline set up, it’s a bit hard to fix the issue quickly.

One trick is to comment out half the site in a binary fashion until it suddenly loads, then keep bisecting until it stops loading. It’s crude but only takes log(n) steps and can be done quickly. You’d have to do it in production, but it might be easier than guessing.