Note that this can be fully mitigated in software: the browser only needs to hide the latency difference, say, by first measuring the upper bound with random pattern and then setting the measured time to that.