I was just curious if I could improve our PHP-based site’s performance. So I attached strace to an Apache process and followed the log of syscalls and counted milliseconds between them. Sure enough, I discovered that 20 ms was being spent each time on a DNS lookup to a statsd metrics collector service over UDP (I remember being told that this was lightweight, since it was UDP). PHP didn’t cache DNS lookups and this was sometimes happening many times per request. I added a static entry in /etc/hosts and the overall latency improved by 30% across all endpoints.

Another hack: I once was consulting for a client who was running Drupal and was going to launch their new site the next day, but suddenly it started crashing on some of the pages. I found out that you can take a core dump of Apache and load it into gdb. Then if you run some gdb macros, you can see the PHP stack trace at the time the crash occurred. Turns it it was some module (tokens?) they had recently enabled which was recursively calling into itself. Not sure why it didn’t hit some stack limit, though. We disabled the module, which fixed it and the client was super happy. If I knew more about Drupal, I probably would have disabled modules in a form of binary search as a first troubleshooting step. But I did know a little about gdb, so that came in handy.