> I understand the problems with adding a random delay to try to add "noise" to the measurements
Actually, this isn't problematic at all... it's called random blinding: https://en.wikipedia.org/wiki/Blinding_(cryptography)
It's a commonly used method used to implement things like RSA in constant time.