Bitsquatting Windows.com | Hacker News Reader

Bitsquatting Windows.com

remyhax.xyz

258 pointsvikrum79 comments

One rather insidious one I'd never considered prior was that lowercase n is one bit from '.'. So you can also bitsquat on things like, say, wwwngoogle.com or mailngoogle.com.

A researcher brought this to my attention years ago with a set of domains in particular I won't name. What was most interesting to me is just how frequent bit errors must happen. According to the research, they'd basically received thousands of emails destined for the correct domain. Really makes you think.

For this bit flip to work it would have to happen before it was handed down to the actual resolver, since on the wire this flip would just cause a invalid parse of the DNS label

This "attack" is against the ordinary user, so yeah is before or at the time of the dns query. I'd expect(probably incorrectly) for the server side stuff like recursive resolvers to have ECC.

So imagine you are sending a mail to jim@mail.example.com. You see it in the address field as correct. You click send, and your client resolves mailnexample.com, gets an address, and delivers it, while you are none the wiser.

This is particularly bad for third level domains, which are more common than you'd expect.

Reminds me of a former colleague of mine who was one of the principal developers behind an Android app, where they could really go to town on making it crash free, since Android offers a phone home function when a crash occurs.

At some point, he made the observation that 1-2% of crashes occur not because of a programmer error or anything, but "chance"; bit flips and the like, in either the app's memory or the phone's services.

So while on an individual basis it's rare, and in practice nobody will really notice an issue, statistically bit flips are significant enough to need attention.

I had a project where we were reaching for the 'cosmic ray' idea. But it turns out it was a supercap that did not hold enough charge on NAND during power off and the way NAND ages. It worked when 'new' but as the device aged and was used it the NAND would slowly take extra time to write a page. Eventually the time was just enough that it would mostly write everything. It would also sometimes randomly flip bits here and there depending on where it was in the write cycle.

Isn't that the reason ECC memory was created?

Yep, and it's super annoying that it's not standard in all devices. I have it on my NAS, but not my PC.

Presumably this shouldn't be a problem for most 'cloud' providers like AWS and Google who run ECC memory, right?

It happens on the system making the initial DNS request: so, for example, your PC.

I don't think bit errors happen nearly as often as people seem to think on HN. Sure they happen but they're relatively rare, especially on good quality servers. We have ECC/hashing all over the place looking for it at various layers. I bet it's 1 in a trillion emails that get a bit error in the addressing if not more. I have written embedded code (FPGA) that ran for YEARs that has hashing and I could probably count on one hand the number of times we've had to address any issue (always failed hardware) and it was caught every time unless SHA512 has failed us.

But what about consumer hardware the might be failing? So your typical 7-10 year old PC still trucking along serving as a Facebook and word machine. Genuinely curious as to how common bit flips are in the wild.

> relatively rare, especially on good quality servers

Good quality servers are the top of the reliability standings. The bottom... the bottom is very far down. Very far.

I'm a hardware guy now. I'm still amazed anything ever works at all. The worst are things like DDR SDRAM that work well enough that people think they can get away without error detection. At least modern MLC NAND flash is so bad that it obviously doesn't work at all without ECC. Better that than silent failures.

The first Guild Wars game actually added a Prime95-style iteration before sending a bug report/crash dump back. An astonishing number of clients failed this test; if I'm remembering the conversation I had correctly, something like half of all oddball crashes got filtered out by that test. Half! (To be fair, overclocked cheap gaming machines may actually be the bottom tier of reliability, but the point stands.)

> I don't think bit errors happen nearly as often as people seem to think on HN.

Back in the days of spinning disks, I had a coworker who had previous worked on the SQL Server test team.

One thing they tracked was bit errors. He said they were surprisingly common to encounter (and of course then account for)

Or, to put it another way, there is a reason modern file systems are paranoid.

Also way back when I managed a huge distributed test cluster, we ran ~3 million tests a week. A non-trivial # of test failures (few hundred) were spurious network failures where the test package didn't get copied to the test client correctly, we detected it "something went wrong" and marked the test as failed. I know in theory Ethernet promises the data gets there without issue, but in reality, at scale, it wasn't. Flaky hardware, bad cable, failing switch, a couple machines with unreliable RAM? Who knows, but it happens.

> I could probably count on one hand the number of times we've had to address any issue (always failed hardware)

Another product I worked on, the bus timings for the DRAM were slightly off. And I do mean slightly, I wasn't directly involved but I think it was something stupid like being 1 cycle short on waiting for something.

We had amazing logging. Soon as we had a few hundred thousand units out, we get ~10 stack traces come in from crash dumps, always in the same place. My team spent a month going over every line of code related to that place in memory.

New build goes out, more failures, again tiny tiny %, you'd only notice this at scale. We track it down, pointer being corrupted, it was stored in the same memory address we had been looking at last time.

Bit was being flipped. We only noticed it because it had hit a pointer. Most of the DRAM, the vast vast majority, was capable of operating 1 cycle out of spec, but some chips weren't.

We only detected and fixed this bit flip issue because the principle engineer on that product insisted that we investigate and fix EVERY single product crash. He insisted on 100% reliability.

Do you think every firmware team for every single component of every single motherboard has that same dedication to quality? Because if not, errors are going to slip through, and they won't even be noticed, not to mention investigated.

Cool writeup. I think some groups or orgs suffer the opposite of survivorship bias, that is, even if an occurrence is rare, they get to see it a lot of times.

> Do you think every firmware team for every single component of every single motherboard has that same dedication to quality?

The answer is, of course, no. Build quality and work conditions vary wildly.

In theory, this should have lower efficacy than bit flips elsewhere in the domain. Periods aren’t actually sent in DNS queries, so you’re decreasing the number of places in memory where the necessary bit flip could occur. The equation changes slightly when you’re trying to intercept HTTPS traffic, though, since a bit flip in a DNS query wouldn’t help you anyway; it’d have to happen elsewhere.

Since you have experience, maybe there's something you can help explain with what was mentioned:

> In fact, out of the 32 valid domain names that are 1-bitflip away from windows.com

"windows" is 7 characters long so how is there 32 1-bitflip combinations?

Also one of the ones mentioned was "windo7s", but "w" is 01111000 and "7" is 0000111, which isn't a single bit flip away unless I'm doing something wrong.

DoD address range is not so mysterious: it’s 11.0.0.0/8, and it’s not been seen on the internet. So it is extremely tempting to take one’s 10.0.0.0/8 and turn it into 10.0.0.0/7.

I would bet this is what is going on there - a large network who decided to take a shortcut with addressing, no evilz haxxorz.

Agreed. I've seen people use assigned (RIR) but not announced (BGP) IP space for years in various ways. Squatting on 11/8 is common to the uneducated.

The buried lede here is what's going on with Baidu. Presumably the initial erroneous request was typed into a browser, does Baidu operate a browser in China? Do they operate a portion of the backbone network infrastructure, and are inspecting traffic?

If not, how could they possibly have identified this as a target for crawling, unless perhaps they are being fed traffic by the Great Firewall?

I don't think they operate a browser anymore. My guess at how this happend was that some user tried to open the site, it failed and the copied and pasted the url into Baidu search which then crawled it, and probably also corrected the misspelling.

I did hear in a talk, where they did this with (as far I remember) google.com, a lot of such wrong traffic came from china too.

Even if they don't have a browser they could have a browser extension

edit: it looks like they also have antivirus software which could be scanning urls

That's easy enough. There are WWW pages that contain "time.wiodows.com". I turned up a couple with Bing, in examples of how to use the ntpdate program, and there are no doubt more, including ones where this is hyperlinked by some "smart" program. No need for a nefarious explanation when a WWW spider found this on a WWW page and decided to crawl it will do just as well. (-:

Even if they're not hyperlinked, at this point, I wouldn't be surprised to learn of a crawler looking for FQDNs and things that just generally look like URLs in text to crawl into.

I don’t buy it. The odds that Baidu only crawls that page immediately after it is connected to by a chinese IP are astronomical.

EDIT: Ah, unless the page containing the link was accessed via a Baidu search. That makes sense.

They could both have been the WWW spider. Search engines have been known to not announce themselves, to check what they are seeing against what everyone else sees.

Oh snap, this was news to MSFT? How did they not register bitflips of windows.com and microsoft.com?

I think I checked this like 10 years ago, and Google had. (well, except the infinite set of subdomains where a dot and an n are one bit away)

And this is why ECC RAM should obviously be the default.

> And this is why ECC RAM should obviously be the default.

For routers and other hardware along the line this would (probably) be too latency-sensitive.

ECC RAM is no slower than regular RAM.

At least, it doesn't have to be. The overlap between overclocked, XMP'd memory and ECC is a null set, but that's not due to physics.

It partially is, the biggest issue with parallel busses like that of DRAM is skew between the bit lanes. You need the electrical pulses to physically arrive at the same time. It's more difficult to layout 71 bits of exactly equal length than 64.

Since you have experience, maybe there's something you can help explain with what was mentioned:

> In fact, out of the 32 valid domain names that are 1-bitflip away from windows.com

"windows" is 7 characters long so how is there 32 1-bitflip combinations?

Also one of the ones mentioned was "windo7s", but "w" is 01111000 and "7" is 0000111, which isn't a single bit flip away unless I'm doing something wrong.

If we use ASCII 7-bit char codes, 'w' is 0111 0111, and '7' is 0011 0111.

7 characters of 7 bits each gives a grand total of 49 flippable positions.

and not all of the flip results are valid in domains, reducing the set. E.g. if 0111 0111 gets flipped to 0111 1111, that's not a valid character.

>X-Forwarded-For that attempts to make the request appear as if it originated from an IP belonging to the US Department of Defense.

Networking newbies always get spooked by this https://blog.erratasec.com/2013/12/dod-address-space-its-not...

Hundreds of ISPs out there utilizing DoD space for their internal addresses.

Interesting link!

> This sort of thing happens a lot. I (or others I trust) have seen 1.0.0.0/24, 22.0.0.0/24, and other instances of 30.0.0.0/24 used this way.

That usage of 1.0.0.0/24 probably did not age well

https://blog.cloudflare.com/fixing-reachability-to-1-1-1-1-g...

I did this years ago with the DNS name for a large CDN (think akadns.net but it wasn't Akamai) and also saw lots of interesting stuff behind it.

Its hard (maybe impossible) to identify identify scale, so I don't know if it was 1%, .1% or .0001% of traffic, but I was seeing hundreds of requests per second.

Since you have experience, maybe there's something you can help explain with what was mentioned:

> In fact, out of the 32 valid domain names that are 1-bitflip away from windows.com

"windows" is 7 characters long so how is there 32 1-bitflip combinations?

Also one of the ones mentioned was "windo7s", but "w" is 01111000 and "7" is 0000111, which isn't a single bit flip away unless I'm doing something wrong.

You can't consider all bit-flip possibilities, you have to consider only those which result in valid domain names. Flipping a bit which results in a w becoming a / will never result in a valid domain name, for example.

Your bit math here is wrong (not sure why).

> "windo7s", but "w" is 01111000 and "7" is 0000111

7 is 055 is 0011,0111

w is 119 is 0111,0111

2nd bit is flipped

McAfee is flagging site as follows URL: https://remyhax.xyz/posts/bitsquatting-windows/ URL Categories: Malicious Sites Reputation: High Risk

I had McAfee flag a restaurant's website because they mentioned alcohol (i.e included their wine menu). I went to the hassle of submitting a request to change the category ... and it was rejected!

The NTP conclusion is wrong.

chrony also uses randomized transmit timestamps:

Transmit Timestamp: Feb 2, 2045 15:47:48.317625828 UTC

As per [0] this is a security feature:

[0] https://chrony.tuxfamily.org/comparison.html

What I'd love to see is a split by which bit is affected and are any versions more popular in the truly benign cases (like ntp).

Although it was confirmed with idle devices that we do see bitflips, I keep wondering if in real world we get more changes due to memory overflows than cosmic rays / failed ram refresh.

The “US DoD” IP could simply be a mobile network operator’s internal IP address leaking via the XFF header. Mobile network operators are known to use public but unadvertised IP ranges such as 25.0.0.0/8 (UK MoD: https://blog.wireshark.org/2010/04/t-mobile-clever-or-insane...) for their internal networks in order to avoid clashes with actual private IPs. For example, by using a “public” IP address, any private IP address requests from a client will route through e.g. WiFi instead of through the cell network. It would not surprise me if a Chinese mobile network operator was using chunks of unadvertised US DoD IP address for the same purpose.

And how many of these would be typos and not bit flips caused by cosmic rays?

You could quite easily test that, by comparing bit-flipped domains to keyboard-neighbour domains.

Or alternatively, just by looking at the distribution of hits to bit-flipped domains (which should be broadly uniform; typos will be non-uniform).

Semi-related: I remember a story from HN a few years ago where someone started experiencing crashes and tracked it down to a single flipped bit in memory which went away when he purged his page cache so the binary was reloaded from disk. Does anyone happen to have a link to it?

I don't remember it on HN, but this is prodding my memory about an article where the specific executable that was corrupted was `bc' on Linux, if that helps any other readers track it down. I think it specifically broke subtraction or something?

I remember it too! This one? https://blogs.oracle.com/linux/attack-of-the-cosmic-rays-v2

That’s the one! Thank you!

> 199,180 NTP Client connections from 626 unique IP addresses

This seems like an abnormally high connections/IP, even if we assume some of the IPs represent multiple clients. Perhaps it's because of retrying as the author does not seem to have sent NTP replies back?

That's over 14 days, so it averages to just under 1 request per IP per hour. That seems reasonable for an NTP server.

What seems stranger to me is the incorrect clock settings on 30% of those computers. Why do they all think it's the end of September (even though the years are drastically different)?

> Why do they all think it's the end of September

I suspect this is just because the author is sorting the text, and the days at the end of September are lexicographically last given that date formatting.

That's why ISO-8601 should be default.

https://news.ycombinator.com/item?id=26341576

But to have your computer bitflip 300 times in a couple weeks is impossible. Either the address is cached incorrectly for a long time or it's just a typo in a config file.

Gonna wager some sort of cheap IoT/embedded device inappropriately abusing Microsoft’s NTP server with a typo in the configuration.

My guess is that for some of these clients they made a typo in their ntp.conf file/NTP config but never noticed it was wrong. I have manually configured hosts/devices in the past for a variety of public NTP servers including time.windows.com.

Somewhat related: I run an http server on the local network for various services. Due to the large number of bogus requests I set up a honeypot vhost which responds only to requests sent without a Host header, i.e. received directly on the IP address. It also logs these requests separately.

It's fun to check the logs once in a while and see all sorts of exploit attempts. Wordpress and PHP seem to attract a lot of attention.

Can you elaborate a bit on how you set things up and introduced some security?

Authorised parties have prior knowledge of the subdomains where the apps reside.

Everyone else hitting the IPs directly (presumably coming from mass IP scans) will be met with a honeypot vhost returning nothing.

An example can be found in the nginx manual with the catch-all approach: https://nginx.org/en/docs/http/server_names.html#miscellaneo...

Fantastic, didn’t know that was possible

As a fun exercise sometime, spin up any cheap EC2 instance in AWS, set up an HTTP server listening on port 80 and 443, and watch your SSH and HTTP logs.

The last time I did this I had random SSH and HTTP requests rolling in within minutes.

I recently interned in cloud security for a mid-sized firm. In my first week, I spun up a free-tier EC2 for some testing (Only port 22 was open) and soon enough, the logs were full of port scans and unauthorized ssh attempts. It is only then I realized the scale at which malicious parties operate.

Suffice it to say, I was very careful with my filters after that incident.

In 2004 a study was done that showed an unprotected PC connected to the internet was attacked within 20 minutes, half the time of a previous study just a few years prior.

If we suppose that it continues to half, say ever five years, it went to within 10 minutes in 2010, within 5 minutes in 2015, and is now somewhere around within 3 minutes for an unprotected PC to be attacked once connected the the internet.

Heck, not just EC2. Try doing this with a home connection. Add RDP into the mix.

It doesn't take long before one scan finds the port, and then a bunch of people start trying different attacks.

The behavior seen here seems to indicate typos more than bit flipping - ex, look at the NTP requests: it works out to roughly 1 request/hr from the ~600 IPs that are hitting them, that's not an in-memory bit-flip.

Certain in memory or on disk bit flips would make it use the wrong domain repeatedly until rebooted, or even until the system was reinstalled.

Having said that, I suspect that if these were investigated in detail you would find software bugs or typos to be the majority. Simple things like a race condition when suspending to RAM could flip a few bits occasionally if the RAM isn't correctly refreshed.

Oh goodness! After domain squatting, one more squatting to think about! Do I have to worry about these things? How likely is it that a solar flare modifies a computer memory right when typing a domain?

The likelihood of your computer experiencing a bit flip at the wrong time is microscopic. The likelihood that some computer, somewhere on the planet at any time? Much higher.

It is far more likely that a DRAM chip will start going bad over time. Unlike solar flares this will definitely cause you problems.

Over 25 years I have experienced this in two different computers.

Unless you live at high altitude. When I lived at 7,200 feet in the mountains of Colorado my ECC system reported bit errors every month.

Anyone using laptops while in flight should be scared.

Or if you live untrustworthy country with nuclear.

https://www.jakepoz.com/debugging-behind-the-iron-curtain/

aren't Ethernet frames CRC'd? Are there other checksums down the call stack, or up the stack at the other end, that would reject such bit flips in the raw data?

Yes, Ethernet frames are CRC'd but some (most?) memory isn't. The web address is bit-flipped in memory, and then the network stack generates a valid Ethernet frame using the already corrupted address.

Wha? Can somebody explain to a non-programmer now one character can "flip" in a computer and become another one? Is this quite common?

It’s extremely common over time, but rare for any one calculation. This is why ECC memory is so useful: https://en.wikipedia.org/wiki/RAM_parity

RAM (the temporary memory where your operating system, programs, etc are stored) stores bits as charge in a capacitor. Now this is where my understanding falls apart, so I'm going to take an educated guess and hope someone smarter corrects me: Random energy from thermal heat or stray cosmic particles can cause these bits to leak from the capacitor or for an extra charge be thrown into the wire as the data gets transmitted.

the use of the department of defense IP there is most likely because it's a ISP using one of the DoD's ipv4 /8 blocks internally. There's more than a few (and not just all in China) that have done that, in an attempt to shovel back the tide of needing to fully migrate to ipv6, or due to lack of other ipv4 resources for unique customer numbering.

I wish I had kids so geeky that this was a regular bedtime story.

Reply on news.ycombinator.com