You did a great job of breaking down the technical complexity, Patrick. You are a good teacher.

"So why did this never show up in development and why did it show up only sporadically in production?" -- I've also run into production bugs that don't manifest in development; your post inspires me to keep as many details as possible in common for the two environments.