My initial hunch was that it was a timing issue, and that I was seeing different behavior based on temperature. Even when I made things super slow to exclude timing, the performance was inconsistent. Next on the list was excluding race conditions (maybe I'm not resetting in the right order, and getting lucky?)
At some point in time, it was 8am after an all nighter after I'd been debugging this, and I hadn't been able to reproduce the bug. Lo and behold, when I give up, the problem starts occuring, just as I open the blinds to get some sun.
Turns out that the display driver had light sensitivity issues. Since it was a cheap display[2], the backside of the driver IC was exposed (the epoxy fill didn't encaspulate it all the way, just the edges; you can see it as the white strip in the Digi-Key picture).
Putting a piece of tape over the IC solved the issue, and I didn't run into problems with the display again.
[1] PCB (business card sized): https://bitbucket.org/cyanoacry/ditch_day/src/3bf75f6bd2fba1...
Hardware picture: http://www.albertgural.com/blog/caltech-ditch-day-2013/image...
[2] http://www.digikey.com/product-detail/en/COG-C144MVGI-08/153...