sleep(3)

We noticed that one line of byte code was crazy hot. We traced it back to the code, and realized it was a special case fallback.

They were summing 10mb of 8 bit encoded integers, in the 99.5th percentile of traffic, so no one noticed. It was using 10% of fleet compute.

We changed the encoding to 64bit; the data size stayed constant, the 99.5%tile latency plummeted, and I got a bonus. Took a week. My contribution to climate change.

A lot of stuff can hide past 99%.