Windows feature that resets system clocks based on random data is wreaking havoc
arstechnica.com
arstechnica.com
Set up a bunch of servers all over the internet with innocuous web pages. Get all of them to include in their SSL headers the exact identical timestamp of July 5th 1998.
Then get the user to connect to all those domains (eg. with a page with a bunch of iframes).
The Secure Time service will see that lots of remote servers all agree with high confidence that the date is July 5th 1998. So the system clock gets set to then.
Then you use a leaked yet expired cert (or maybe one signed with a broken algorithm) to impersonate an update server or steal valuable cookies/tokens.
Expired certs typically fall out the back of revocation systems too - so it really is just the expiry date/time that protects against their use.
It seems to me that it could still be used to bring down a windows server right around the time that you wanted to, which is still a potentially serious security concern.
I can feel an experiment coming on: Mint an OpenSSL based CA and use it to generate 200 certs for randomly generated CNs. The script could write out a zone file and web server vhost configs. Pop the CA cert in a Win PC/Server trust store. "Fix" the time on the web server. AutoIT could be used to poke a browser at each vhost or an iframe monstrosity.
I have yet to see this snag myself but it sounds like an MS style screw up. Excellent engineering fucked up by one wrong assumption which leads to a huge towering monstrosity. Sheer arrogance precludes a back down, MS ploughs on and then after a year or two, a fix is silently released and the victim shaming carries on for a little while longer. Meanwhile social.ms carries on advising sfc /scannow and then reinstalling the OS.
I really hate that. If there is some problem, properly diagnose it, figure out how it happened, and figure out how to change things so it can never happen to you or anyone else again.
Or you could just wipe all config and reinstall everything and hope it doesn't happen again!
sfc /scannow can fix issues but it is not a panacea and certainly not the first thing you should reach for. It sorts some aspects of .dll hell but that is an app thing and a complex one.
As you say, get the logs out and your stethoscope. Sadly most of us don't have access to source with Windows, so we rely on intuition, 30 years of experience and some really cunning search queries. Even though Google search is largely wank, a decent and carefully curated ublocklist can winnow out the worst rubbish.
Having said that, Outlook is totally damned when it comes to logs. Outlook trace logging is an Outlook programmers debugging tool only and of absolutely no use for a sysadmin. You literally fly blind with Outlook and are best off looking at Exchange logs instead. The Outlook devs are only interested in themselves and no one else. You do get CTRL - r click on Outlook icon to unlock the connectivity checker to give you absolutely no help apart from showing how bad things are. The autodiscovery checker is handy and I have fixed configuration issues with it.
Wireshark simply listens. If you give it the private key for a TLS exchange it is able to decrypt the stream.
Fiddler makes changes to the stream - it turns a single stream into two streams with the Fiddler in the middle, Wireshark does not - per se - it just listens but because it will be a tap in a stream it will slightly affect that stream, unless you use a switch that can mirror ports - and that's what I do.
When it is your web server (int al) then you have access to the private key and then Wireshark's minimal touch is indicated. Otherwise Fiddler or similar (Squid and others for me) is indicated instead.
The GP likely doesn't have the private key if they're using Outlook. As if they're using Outlook it is most likely with Exchange. And if they're using Exchange, it is most likely EXO. Hence, Fiddler as the appropriate recommendation for this portion of the thread.
Port mirroring/SPAN/RSPAN is the only way to be an observer without affecting the traffic you are inspecting. Fiddler literally does a man in the middle attack.
If you don't have access to a private key for a TLS exchange and stream, then you cannot do quite a lot of diags.
In the end, Fiddler and the like might do the job but to really get close to the wire requires full knowledge and a much lighter touch. That will be a port SPAN/mirror and Wireshark and a private key.
If you're a big enough customer, and you contrive a way to rely on this bug, then you've hacked the Microsoft backwards compatibility commitment. They'll have to preserve the functionality in perpetuum.
I don't recall the exact conditions that determine this but troubleshooting it is an exercise in thinking you're having a stroke.
/etc/systemd/timesyncd.conf
NTP=w.x.y.z
/etc/ntp.conf
server w.x.y.z iburst
That doesn't sound right. Are you using some kind of externally controlled list of round-robin servers?
(A smarter time daemon like chrony would've kept track of the RTC drift.)
This is the installer I used to use: https://github.com/Gictorbit/photoshopCClinux
This one is newer and looks even more promising: https://github.com/LinSoftWin/Photoshop-CC2022-Linux
Their plant was 2 hours away, but I was driving past there to visit a friend, so I made arrangements to drop by and investigate.
For reasons unknown, most of their Windows desktops were configured to a year in the 21st century, probably 2096 instead of 1996 but it’s been a while. Wish I knew how they ended up that way, but I’m surprised dialup was the only thing they noticed that was broken.
To address the issue they were trying to address (what happens when a mission critical server's RTC malfunctions or the battery dies?), Windows should treat it no differently than any other hardware malfunction in a mission-critical server: raise an alarm so a system operator can take a look and address the issue.
> “The false assumption is that most SSL implementations return the server time,” Simen said. “This was probably true in a Microsoft-only ecosystem back when they implemented it, but at that time [when STS was introduced], OpenSSL was already sending random data instead.”
And they link the reason for sending random data at https://mailarchive.ietf.org/arch/msg/tls/_clS-TIIlZUcid_2S4...
The post starts with
> Here's something I wrote about removing a fingerprinting opportunity from the current TLS protocol.
So the point is not that it was a bad idea, it is that MS didn't keep up with the world and didn't respond to requests from their customers to fix the problem.
I think that if you're making mission-critical systems rely on nonguaranteed behavior from systems you don't control, then it's a Bad Idea.
https://www.ietf.org/archive/id/draft-mathewson-no-gmtunixti...
> “Since the device is not in a state to communicate securely over the network, it cannot obtain time securely over the network as well, unless you choose to ignore network security or at least punch some holes into it by making exceptions.”
So, because they can't trust the network, or the time, they instead choose to randomly trust some of the requests over the network.
But it's okay because they rely on the certificate from that request to report whether it has been revoked or not.
It really feels like someone's thought experiment that was cargo-culted into production. Timestamps are truthy to begin with, and mission-critical systems are designed to die and be reborn anew like the common Phoenix. Physical issues that might come up are addressed in the spec and logical issues stop everything they touch. At no point have I ever hoped that a third party would start pumping garbage into the problem.
(Whatever other representations there are of "The Good Idea Fairy", I can't help but think of Baldrick and his "cunning plans".)
Or just accept that absent NTP, w32time maybe just shouldn't try to set the clock to whatever a circus clown tells it?
This sort of reminds me how there is a buried registry setting to tell w32time to not place the firmware clock at very high stratum in the time sources list which also is just a giant WTF in how those two behaviors were decided on.
On the other hand, with virtualization, who knows how long it's really been between clock ticks. Let's say someone suspends a VM for a couple months and then unsuspends it; hopefully the clock is synchronized, but maybe it isn't, and you do need to jump ahead.
I thought I saw in the article that there are failsafes, but the allowed range is still pretty broad. But reporting is going to tend to be dominated by larger values, because smalller jumps are less likely to be noticed or more likely to be considered an accidental change.
But since the vast majority of them do not fail in that matter if the normal, configured methods of determining time aren't working, using TLS parameters that have been deliberately randomized by standard for like what, a decade now?, can not possibly be a more straightforward failure mode than logging errors and waiting for operator input.
If it's that critically important, embed a fixed signature for a set of known time sources and query those as a last resort. Microsoft certainly has the resources to set up a few to do this the much more obvious and predictable way.
Yes, that gets annoying fast for said user, but a good incentive to eg fix the bios battery.
/ sarcasm
But that network is not trusted.
Imagine this: You boot a machine for the first time, and the system clock tells you it's January 1, 1970. You might know when your OS was built, so you could maybe hard-code some sanity checks there, but you basically don't know what the date is.
You want to communicate securely with weather.gov? Sure, you can do that over SSL/TLS. You send it a list of cipher suites and SSL/TLS versions you can use, the weather.gov server selects a cipher and version, and sends you their certificate. It's valid from Jun 2023 to Jun 2024, signed by a DigiCert TLS RSA SHA256 2020 CA1 certificate valid Apr 2021 to Jun 2024, which is itself signed by the Digicert root global CA encoded into your OS's root certificate store, valid Nov 2006 to Sun, Apr 2031.
Wait a second. Or maybe 1.7 billion seconds. Are any of those certificates valid? Do you want to let the attacker set the date back to when a revoked, potentially leaked Trustico or Symantec private certificate was valid?
You don't have a secure network if you don't know what time it is.
If all available time sources are not trustable and none of the existing answers make sense, do not try to set the clock! Coming up with more complicated ways to use untrustable data is not an improvement.
It definitely applies even more to the data it is trusting in this case.
The alternative is to pretend that you can somehow divine a 'projected secure time.' Just, the gall to even write a sentence like that.
There's no way for the server to accurately determine the client's hardware clock, barring some sort of user-hostile hardware clock with a nuclear battery or hardened GPS or radio time receiver or something.
It's 5 o'clock somewhere!
> Because Secure Time Seeding used SSL certificates Windows already stored locally, it could ensure that the machine was securely connected to the remote server. The mechanism, Microsoft engineers wrote, “helped us to break the cyclical dependency between client system time and security keys, including SSL certificates.”
But in that case, why does the Windows time service connect to a random server running OpenSSL instead of a trusted server under Microsoft's control like time.microsoft.com (or whatever)?
"Golly it's getting expensive hosting 100 billion random requests per year, lets let someone else shoulder the problem".
While this might not be it, we've seen plenty of other providers really screw up with time like this. I believe it was Linksys years ago just pointed their NTP as someone elses NTP servers and flooded them out.
Yes, that's a very good reason not to trust the network to tell you what time it is.
Unfortunately, Windows, because it doesn't trust the network in the broad sense, relies on specific and known-to-be-unreliable SSL handshake data on the network to reset the clock.
My guess is that it was created because of Windows Phone 10. But it's not a feature I'd like even on my phone.
In any case, it seems like a poor solution especially given that random timestamps can occasionally look valid (they’re random!).
The problem they’re trying to solve seems to be to obtain a trustworthy timestamp in the presence of a potentially hostile network - one that could be presenting expired, cracked certificates for every connection to arbitrarily tamper with all SSL connections. It’s not a trivial problem, for sure, but resetting perfectly valid clocks using this algorithm seems like a strict downgrade….
sudo ntpdate time.windows.com
Because my WSL VM drifted apparently 29000 seconds into the past for...reasons... causing the az cli login to fail. With an ugly traceback, of course.
The story takes place in a future African megacity before, during, and after an alien invasion. Part of the game's story is shown from an AI's perspective who controls the city's operations trying to evacuate the Human survivors.
Whenever the story is shown through the AI's perspective, you can see an overlay of the AI's working memory including which suburb's cameras are being monitored, the emotional state of the AI, and the current running timestamp.
Some of the aliens end up withdrawing from within the city by making an FTL jump which causes extreme localised time dilation.
Immediately afterwards, the timestamp in the AI's working memory is frozen because it's not sure what the exact timestamp is anymore, each system it connects to gives it a slightly different time depending on how close it was to the FTL event.
It guesses a likely time and uses that until sometime later it is able to reconnect with other unaffected human networks and resumes the correct timestamps.
This time heuristic was shown in the game, but also makes an appearance of sorts in the announcement trailer (seen in the bottom right): https://www.youtube.com/watch?v=WroxHMo6B_k
One of the annoying consequences is that some parts of the system decide to clean up "old" data. Surely data that has been stale for 50 years can be deleted, right? I cannot imagine the impact of something similar happening on a production server.
$ TZ=UTC date --date=@$[$(date +%s)*2]
Wed 31 Mar 15:33:50 UTC 2077Perhaps something that is meant to add a delta to a timestamp accidentally adds 2 timestamps together.
This shouldn't be overkill for a server dedicated to timekeeping on a large company network, right?
In a datacenter one could even go for a matchbox sized atomic clock. Last I checked (years ago) such a miracle device could be had for ~$1500.
Then: how often does atomic clock (or even TXCO) fail, in practice? My guess: only about as often as even the backup power fails.
Note that either should provide accurate timekeeping without even going outside a physical location. That is: without using 'random' NTP server on the internet, GPS or whatever.
Most atomic clocks will have 20+ year lifetimes. OCXOs age out a good bit faster, but they're rather cheap (<$200 for ones good enough to provide holdover for a GPS module). TCXOs don't age as quickly (they're not literally in an oven), I'm not sure how long they can be expected to last.
Even when using an NTP server to set your time, network latencies have to be taken into account. Otherwise, if the request took 300ms round-trip and that weren't accounted for then your machine is already 300ms off from "official" time. Depending upon what you're doing, being that far off can be critical.
These are things your dollar-store quartz watch simply doesn't need to concern itself with.
If you have a critical need for leap second time you're typically using your own means of NTP and not Microsofts broken ass bullshit.
I love that statement! It reflects my 40 years experience in software - if you're doing something serious then don't use Microsoft's broken ass bullshit. I smile and remember Microsoft's origins as a gaming company. To this day that's what they're best for!
Does it? My system/bios time is in UTC. The timezone is not managed for me, when I travel I change it. DST and leap seconds are deltas you add to UNIX epoch. Tracking the number of ticks since epoch is a simpler task, much closer to what a quartz watch attempts to do on its own. Both fail when the battery does, of course.
If your end user facing application can't handle 300ms drift that seems like a bug in the application. Even stuff like TOTP generators let you apply a time delta correction and servers usually allow >1 minute drift.
If your quartz watch battery goes flat you replace it straight away because a watch with a flat battery is 100% useless. But if the battery on your motherboard goes flat you don't even notice, so why would you replace it? To a layperson, "my computer's time is always wrong" is not likely to be caused by a random flat battery that they didn't know existed.
Microsoft came up with this "Secure Time" system so that even computers with flat batteries would be able to work out what time it is, but obviously it is kind of weird and sometimes makes wrong decisions.
There are longer life batteries, but they're _expensive_, e.g. a Tadiran TL-4934, 1Ah, with a shelf life of 40 years will cost ~5$ in bulk.
"Microsoft came up with an internet boot image that allows your computers to boot windows even if the hard drive is dead without users realizing this is the case"
Honestly, when the battery dies, it's time to take it to the shop.
I've seen other services that use https to bootstrap time in case other clocks are unavailable or suspect (or use a limited expiration certificate to authenticate!), it's a bit difficult because you have to ignore or postpone checking certificate expiration when validating the x.509 certificates, parse the http date header, and then presumably check that the date provided is within the time the certificates involved are valid.
[1] https://datatracker.ietf.org/doc/html/draft-mathewson-no-gmt...
(*) I suppose this does not work as well on Mars. I suspect that the time on the various extra-planetary vehicles is tied to earth time but would not work so well if Mars were ever colonized.
Since it's fun to imagine (and then find flaws in proposals) here's an idea for y'all:
1. If the device reaches out to a dozen servers at different usually-trustable domains (e.g. microsoft.com, nist.gov) and they _all_ have "expired" certificates, that becomes a sign that My Local Time Is Unreliable. (Or that the machine is being booted up by an attacker who has connected it to a fake micro-internet.)
2. The Windows device contains a cert or public key which does not auto-expire based on clock-time, but is only replaced as a side-effect of OS patches. This cert/key exists for one specific purpose, allowing it to securely get a time from an MS time fallback server. Such a server would not normally be queried, and can have very relaxed performance/accuracy/uptime requirements. It could literally just be a GET call returning a epoch integer.
3. This way the client system can bootstrap up to something close-enough-to-current that the regular NTP process can be completed for a more-precise time.
Obviously this can fail if the attacker injects a fake cert/key into the machine... but at that point isn't it Game Over anyway? The same attacker could just put in their own certificate authority for any service.
It may not work for computers that are left unplugged and unpatched for a few years, but it sounds like the real issue are live servers, or ones that just suffered some power-supply hiccup.
Set-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Services\W32Time\Config" -Name "UtilizeSslTimeData" -Type DWord -Value 0
Reload using:
W32tm.exe /config /update
All of these work-arounds for unneeded hacks (like in the article) drives my temper way up. Just do the simple thing in kernel space or the OS level. No suprises.
Under ordinary conditions, the clock discipline gradually slews the clock to the correct time, so that the time is effectively continuous and never stepped forward or backward. If, due to extreme network congestion, an offset spike exceeds the step threshold, by default 128 ms, the spike is discarded. However, if offset spikes greater than the step threshold persist for an interval more than the stepout threshold, by default 300 s, the system clock is stepped to the correct time.
Let me guess, time isn't set, so dnssec is broken, so ntp servers won't reolve, so you can't set the time?
So lame, ntp servers shouldn't be using dnssec.
Anyway, STS as described is an obviously broken protocol. It’s not clear to me that it is even capable of reliably getting a rough time.
So far, it all makes sense. It stops making sense when their attempt at breaking the catch-22 seems to do so by in fact ditching the "secure connection" requirement, just doing so in an obfuscated manner.
I agree, if you already violate your own rule, you could have saved everyone a lot of hassle and just pull the time from a plaintext HTTP server.
But then you might just as well ask a plaintext NTP server and go back to where everything started...
I don’t really know why bootstrapping over HTTP then adjusting via secure NTP is better than bootstrapping via plaintext NTP then confirming with SSL enabled, but I guess it at least means two servers and protocols would have to be compromised.
Our TAM is like a funeral director. Always expressing his deep sorrow.
Why then are the first lines of the STS code not something like
// server receives reliable timekeeping through the Network Time Protocol.
if (ntp_enabled() && ntp_working())
return 0;https://learn.microsoft.com/en-us/windows-hardware/design/mi...
anything earlier than 8th gen core? fuck you
I’ve been running Linux since 1990s. It’s best it ever was with the HW support, but saying it’s not an issue is just a straight up lie.
I think MS should improve implementation in two ways: - ask known trusted sources (like their own servers) for SSL timestamp - use that timestamp only internally (as opposed to system-wide) to acquire valid NTP timestamp
Windows 7 is still more reliable, but with the updates being over, people are forced to upgrade.
But Windows 11 is less reliable than popular Linux Desktops. There might be an exception if you have NVIDIA, but I'd say less than a few percent of people have a video card.
* broken on arrival and never really fixed
* negligently implemented that causes serious harm
* a dark pattern that no one actually wants but improves the windows margins through covert means (data harvesting, embedded adware, etc)
I’m sure they exist, but the prevalence of the badness is so striking.
Look, you can use (with some care) time from a Kerberos KDC for the domain/realm to which the host is joined if you like, but you can't use the time from random application servers!
What were they thinking?!
gps (and friends) is a nice way to get access to a very accurate timesource, but it's not always worth the cost.
Here's the PowerShell cmd to do so: Set-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Services\W32Time\Config" -Name "UtilizeSslTimeData" -Type DWord -Value 0