You're exactly correct. This is why the browsers decreased timing resolution in javascript so that you couldn't time memory accesses accurately enough to tell if the address was cached or not.
What does that do, besides turn the exfiltration problem from an immediate one into a statistical one?
Maybe there will end up being a new Jumping Around Kernal Address Space System (JAKASS - a cousin of Linux's FUKWIT patch) that periodically resets kernel ASRL to make it fully impossible.
The paper they linked to references this one: https://www.usenix.org/system/files/conference/usenixsecurit...
I think this is what all sandboxes have to do: set the TSC disable flag, restrict system timer precision (make it configurable per sandbox: web servers generally don't need more than 1ms precision), make system timer report fuzzy (randomized) time. Heck, why not also make the CPU run at randomized frequency to mess with busy loop timers.
Consider an Olympic 100 metre sprinter. Today we time this event very accurately, I think it's to one hundredth of a second, using sophisticated technology.
But even if the judges used a much less accurate mechanical stopwatch, Usain Bolt wouldn't actually be slower, we'd just be less confident of how ridiculously fast he is.
In some special cases, timing things very accurately might be essential to a use of Javascript, but I can't think of any examples off the top of my head.