OP here. Your project is really cool, thanks for sharing. I was considering the same idea, but it means updating the drivers to work on an ESP32

The best option I found so far is a timer shield that can wake up the Raspberry periodically: https://www.pishop.us/product/sleepy-pi-2-power-management-s...