As I start asking if anyone has any idea, someone else mentions they've seen that before too, many weeks ago, but then it "fixed itself".. Today was the 1st, and when I looked at the previous failure for this test, it was also on the 1st, but it was a couple months ago. Last month the tests were passing on the 1st. So I start to dig into it.
To put this in perspective, we had dozens of other tests that were checking the same calculation for various time spans across months (including explicitly for months with both 30 and 31 days, and February-to-March for both leap and non-leap years), and all sorts of combinations of different values, missing data, etc, that had mostly been there for well over a year. We had actually spent a fairly significant amount of time thinking about how to test this for all the different combinations of dates and data.
As it was the 1st, one of the other tests was actually running with the exact same start and end date/times as this "yesterday" test, but was passing. So I started looking at the mock data each was using.
Turns out it was in fact a legitimate bug, but only happened if it was currently the 1st, AND yesterday was not the 31st, AND there were no values at all for the current month (or anytime later).
The mock data for the "yesterday" test didn't have any values after whatever yesterday was. The explicit date test for 31st-to-1st happened to have a 0 value on the 1st, which meant this bug didn't happen.
Mostly out of curiosity, I looked further back in test history. There were in fact 3 or 4 separate times this test failed, all of which were on the 1st of either March, May, July, October or December. But not all of those dates -- because sometimes the 1st was on a weekend, or just no code was pushed, and no build was run.
This was also in production, but probably was never seen by any customers (none had reported it) because the "yesterday" value was only ever displayed in the UI, and most of the time data is added hourly, so by the time a user logged in on the 1st (say, 8 am), it was almost certain there was some piece of data added (even if it was 0).
We added an explicit test with the data for this situation, and of course fixed the bug.
However, this will live on as by far the most obscure time-based test failure I've ever had to deal with.