That's true, but any reasonable datacenter operation is going to detect clock skew beyond some tolerance (order of seconds) and either correct it, or decomm the nodes until it can be corrected. Pragmatically, data systems like Roshi have to operate within that tolerance, but don't need to accommodate huge differentials. Or, maybe more accurately stated: it's OK if they have weird apparent user behavior in those circumstances, as long as it doesn't throw the whole system into a logically nondeterministic state. Which is the case here.
> If the client timestamps come from your web servers, and they're all on NTP, then it won't be too bad, and the effect will only happen for the maximum NTP offset.
That's right. And it's important to zoom out and look at the actual consequences of missed writes at the application layer. This stuff only comes into play when a specific user makes different actions on a specific key, i.e. if a SoundCloud user likes and then immediately un-likes a track. It could be that the like "beats" the un-like, due to clock skew, and when the user reloads the stream, they still see the track with a heart next to it. "That's annoying," they say, and un-like it again, which almost certainly works. No big deal.