Hacker News Reader

Comment by fjni | Hacker News Reader

> I already knew from previous experience that for some reason, AT&T traffic to fast.com is throttled. Why AT&T wants bandwidth to appear lower than reality is a mystery to me, but I digress

This I think is because they throttle ip addresses for known video streaming sites. That is one of (the only reliable) ways an ISP can get the streaming provider to drop the stream to a lower quality one by default. Since fast.com is a Netflix ip, and the isp can’t distinguish whether it’s video that is being transferred or a file to measure throughout, the speed test gets caught up in it. Said the other way around: fast.com is great to see actual throughput from Netflix as opposed to some fake throughout from a dedicated speed test site for the exact same reason.

View on HN

> Said the other way around: fast.com is great to see actual throughput from Netflix as opposed to some fake throughout from a dedicated speed test site for the exact same reason.

Yup. This is pretty much the reason Netflix created fast.com. They wanted a speed test service that couldn’t be gamed by ISPs. Many ISPs will prioritise traffic to know speed test services (like Ookla’s speedtest.net), making their services appear faster than they’re under more normal usage.

By placing fast.com on Netflix IPs, ISPs either have to prioritise all Netflix traffic (which they’re very unlikely to do), or accept that fast.com is going to provide a more realistic measurement of their performance.

Now that my ISP bundles Netflix subscription into their internet plans, access to netflix and fast.com now practically saturate the fiber link, while before it was outright blocked. Hooray for no net neutrality I guess.

Another fun part: when netflix IPs was blocked by this ISP, it's pretty much impossible to use netflix because the only way to get around the block was to use VPN, but netflix itself blocks VPN access.

I'm curious where you live and what provider does this weird Netflix reselling practice.

Streaming services do this all the time in Asia. Instead of paying with credit card, you pay via your carrier/ISP, sometimes at discounted pricing. I paid Netflix, Disney+, Amazon Prime and Zoom this way.

Common in America (e.g. https://www.t-mobile.com/tv-streaming/netflix-on-us) and the UK (e.g. https://news.virginmediao2.co.uk/virgin-media-o2-revamps-top...)

> Hooray for no net neutrality I guess

Surely it would essentially saturate your fibre if there was net neutrality, unless you don't pay for full fibre speeds?

Parent was being facetious.

What is the internet anyway? Are there truth in advertising issues around selling internet access with excessive filtering?

Your ISP entirely blocked netflix? That's incredible shitty.

Early in the pandemic, I spent a while without wired internet using a wireless hotspot from the library which would not connect to Netflix but any other streaming video service was fine. I forget who the wireless vendor behind the hotspot was—I think it might have been Verizon.

So there's a bit of a weird fact about public libraries, which is that it probably wasn't on the normal internet (Internet1) but was in fact on Internet2 (see [1]).

Internet2 is used by hospitals, among other things, and as such has higher robustness requirements than Internet1. The pandemic created much higher demand for bandwidth from hospitals, forcing Internet2 providers to scramble to keep up in areas. As such, blocking high-bandwidth sites which are clearly lower priority than medical traffic might have actually been a reasonable move.

[1] https://en.wikipedia.org/wiki/Internet2

Not quite sure I understand - what do hospitals need a ton of bandwidth for? Why would those bandwidth requirements rise significantly during the pandemic. Sure there were a bunch of people on vents in the MICU, but pretty much every elective procedural service plummeted.

I'd imagine telehealth played a fairly large roll. White crispy video calls are not that demanding individually 10s to 100s of them at scale per location (and an exponential increase of calls in general) might be something more of a problem for that sort of infrastructure.

There are very few medical centers where there are 100s of simultaneous clinic visits. During the pandemic a lot of providers did home telehealth. Even 100s of simultaneous telemedicine visits don’t take much bandwidth compared to HD video streaming. Also from a QoS perspective - these are still terminated on the public Internet.

It might have been difficult relative to the infrastructure at some hospitals - but that is to my point that hospitals in general are not very demanding bandwidth wise and I would be a bit surprised they’re more than a rounding error on the Internet2 in general - especially compared to all the academic research involving large models and just video streaming in general that blew up during the pandemic.

I’m curious if the GP was speaking from some firsthand experience or is just making some conjecture.

Other than telehealth, we also use the bandwidth for remote desktops/apps and lossless high-resolution radiographic images. While one might expect this traffic to occur over local networks, many hospitals have multiple sites and partnerships that are connected over the internet. So, for example, one might operate the hospital's apps while sitting in the library at the affiliated university.

> So, for example, one might operate the hospital's apps while sitting in the library at the affiliated university.

Remote Desktop and even PACS are not huge bandwidth consumers in this day and age of 4K streaming. At my large academic institution usually only the radiologists use the PACS client directly on workstations - all the other clinicians are using a “zero footprint” viewer over Citrix like everything else.

That "zero footprint" viewer over Citrix is susprisingly resource-intensive, even at 1080p with lossy compression. If everyone used it (say, if every workstation were a Citrix client), it quickly adds up. During peak hours, it positively chugs. Not sure if networking or compute is the bottleneck here. Perhaps you're right: some institutions have not caught up to this day and age of 4K streaming, and their pre-Netflix infrastructure is buckling under the weight of all these Citrix streams.

I've worked at two hospitals. Both of them "had" Internet2. I was excited, as I had not heard about it in ~20 years.

Neither of them actually ran any traffic over it.

Heh, just revert back to pirating. That’s what I do whenever things don’t work.

From the above thread it looks as if piracy might actually be a more bandwidth economical option than streaming services. Spread the load of the bandwidth around a bit more. Download any time other than peak streaming hours to then watch, self-contained, during peak streaming hours.

Plus you can rewatch without downloading again. And use sneakernets for sharing with your friends.

I've always disliked how wasteful streaming is in respect bandwidth consumption. However, the "efficiency be damned, just make it fasterer" attitude towards bandwidth is, if not the biggest, then definitely among the biggest, drivers behind the ever increasing internet throughput.

Thinking about it, it seems like watching habits could make streaming be more or less "efficient" compared to downloading once and storing it locally (depends on how one chooses to calculate efficiency tbh).

People that have things they regularly re-watch are obviously gonna benefit from having a local copy they can access entirely on their own terms.

It hadn't occurred to me until recently, but the way I watch stuff has me going through a pretty large amount of data yet I'm also excessively unlikely I re-watch nearly anything that I've seen in the previous 2 years. Whether something is streamed or stored local, it has to be downloaded the first time it's watched. If it'll take 2 years before I re-watch something, it's a waste of time and money to keep a backlog of multiple disks worth of video

With music it's quite different though, I often have the same albums on rotation and it's great to be able to cache them all locally.

Couldn't agree more. I've stopped paying to stream music entirely and now pay to own FLAC files which I store and use on my PC and phone. Almost like regressing back to having an iPod, and I personally love it.

I never gave up my music files, and never will trust service providers enough to do so.

Huh. I typically watch the same thing over and over repeatedly for like a year or more.

Very wasteful if I did much streaming.

Spectrum used to dump all Netflix traffic to a device on their regional network. If you blocked that IP range, the service would perform dramatically better.

> Now that my ISP bundles Netflix subscription into their internet plans, access to netflix and fast.com now practically saturate the fiber link

They may have an ISP-local netflix cache, from Netflix themselves not some home-grown hack, so they can achieve that with some reliability without it costing as much as it otherwise would for bandwidth peering.

Netflix distributes "red boxes" to ISPs which are exactly that: edge content caches.

(PDF): https://openconnect.netflix.com/Open-Connect-Briefing-Paper....

Mine did the same and charged for bundles of HBO Max, Netflix, Disney+ Until recently they reverted it after bunch of complains and tonnes of people changing their ISP

Netflix blocks datacenter VPNs but you can use residential VPNs (which tbf are more expensive) to circumvent this.

hmmm, IMHO, traffic for video and everything else ought to separated into 2 classes. Video obvious takes much much more bandwidth than other stuff. It would be good to allow none video, smaller assets that are needed to to make a website function to have priority over video files.

BTW, AT&T has terrible records for dropping customer's traffic and also email without warning or notices to their customers. Why such terrible company is around amazing how corporation don't really survive by merit in our current world.

Couldn't ISPs just sniff the SNI hostname to differentiate fast.com vs actual Netflix video streaming?

The fast.com page kicks off requests to nflxvideo.net domains for the actual speed measurement. And it wouldn't surprise me if actual Netflix video streaming made occasional connections to fast.com purely to make it harder for ISPs to cheat.

You can think of the requests to fast.com as just loading the speed test control scripts and user interface. The actual speed test loads files from the same servers (with the same SNI hostname) used by actual Netflix video streaming. It wouldn't surprise me at all if the fast.com speed test loads real streaming video segments from these servers, the only difference being that it doesn't have the decryption key for these videos.

That's surely how I would have done it.

It would also not surprise me if dns requests for fast.com temporarily elevated bandwidth limits for netflix

It does. You're loading segments of real movies on Netflix that are not encumbered by copyright.

But they're intentionally otherwise indistinguishable from any other real movie on Netflix.

Yes a few ISPs actually do this (not just for Netflix, but for other sites in general).

In the case of Ookla's speedtest, many ISPs host speed test servers to eliminate any variability of the wider internet. That's why it may seem they are prioritizing it. It's not a useless tool either as lets you narrow issues down to either your network or ISP.

It definitely looks like testing sites are prioritized. The fastest download speed that I have gotten is maybe 7 MB (bytes) per second; generally it is 2-5 MB per second. The speed test sites generally get 100 Mb per second dowload. In general the best I seem to get is about half the speed of the speed test sites. To me the real speed of the ISP is how fast one can download something one wants, not the result of a test. I would prefer to see results for downloads/uploads from youtube and various CDN networks and popular sites. I would also like to see ISP have a URL that is inside their network to test upload and download so that one can at least isolate what part of the connection might be lagging. Actually, I just used devtools to snag a 25MB file from fast.com. Curl/wget gives a speed of about 3 or 4 MB per second. That does not really seem to match up with fast.com download speed of 70Mb/second. 70/8 is 8.75, which is about double. Is fast.com accurate? Is my math wrong?

I personally like https://speed.cloudflare.com since it just looks like you're doing typical CloudFlare traffic. The results viewer is also quite nice.

My ISP (Telecom) gives me a speed of 250Mbps using fast.com and 40Mbps using Cloudflare's tool.

Your isp likely has a Netflix oca or is directly peered with Netflix directly. It’s standard for any decent sized ISP since it’s either the largest or 2nd largest by volume peer (youtube being in the other).

Very cool! I never knew about this! I really appreciate the latency during upload test. I've bookmarked this and will stop using speedtest.net. Thank you!

MB = megabyte

Mb = megabits

1 byte = 8-bit

For years, AT&T sales reps would refer to MB when they meant Mb. Soon after I started service with them, I called to ask about the promised speed, and the tech insisted they sold Mb/s. He conferenced in a salesperson and was embarrassed to discover that the salesperson talked exclusively in terms of MB/s.

Did they just spell it with the abbreviations? Or did they pronounce 'megabytes' deliberately?

They literally said it out loud. I had spent the last 7 years on a fast university network, and had never paid for internet before that. I honestly believed that they were going to deliver the speeds they claimed.

I wouldn't have believed "megabytes" was an Internet speed, because it never is. It's literally impossible for that to be a signaling rate on any known Layer 2 technology. ISPs use Gibibytes or Tebibytes (which they misname as Gigabytes/Terabytes) for transfer caps, but that's apples and oranges.

> I wouldn't have believed "megabytes" was an Internet speed

Perhaps HNers know better, but among lay people, pretty much no one talks about megabits. For example, people know that an MP3 is about 5MB. They know that they can email attachments up to 20MB (or sometimes 10). But I've literally never heard a lay person talk about bits, or bits per second.

And the fact that ISP's transfer caps are denominated in bytes makes things all the more confusing.

There was a time when everyone considered a kilobyte to be 1024 bytes, a megabyte to be 1024 kilobytes, and so forth. The main reason this changed is because disk manufacturers advertised e.g. 1 billion byte hard disks as having “1 GB” of capacity and, to pretend they weren’t ripping us all off, proceeded to claim that the definitions were ambiguous. But even today, if you go out and buy an 8 GB DIMM, you are getting 8 * 1024 * 1024 * 1024 bytes.

(As a historical side note, there was even a weird transitional period where a “1.44 MB 3.5 floppy” held 1.44 * 1000 * 1024 bytes, or what everyone at the time would have called 1440 kilobytes.)

Personally I think it’s unfortunate that standards bodies and even courts have sided with the disk manufacturers. I also find the SI binary prefixes utterly revolting on an aesthetic level, so I’d much rather join whatever parts of the tech industry are still holding out.

I was trying to ask if there was something more that the speed test do other than multiple or divide by 8. Is some other overhead that they add in? If not, then their math or testing seem to be off or curl/wget seem to be different than they get via the browser's javascript engine. To me it seems that the speed test number is inflated or higher than what one will even for the same URL off of netflix or cloudflares URLs used in their speed tests.

On my ISP (Three UK), a DNS lookup to fast.com gives you faster internet for the next minute or so.

Therefore, I have a background bash loop just looking up fast.com every 30 seconds all day long.

Out of interest, which Three plan (& APN) are you on?

I’m working on using a Smarty (Three UK-owned budget label MVNO, for those unaware) SIM card in a 5G modem as a backup line, and haven’t been super impressed with speeds — so curious about trying your tip.

> Since fast.com is a Netflix ip, and the isp can’t distinguish whether it’s video that is being transferred or a file to measure throughout,

It is really trivial to do basic traffic snooping and see what people are looking at. I'm surprised it isn't more common.

I figured it would be harder, or perform worse, but I easily wrote a little piece of software that filters the TLS ClientHello for arbitrary domains. Maybe 10 years ago hardware wouldn't have been able to do this, but I bet it's no big deal now. So your filter chain just looks like <Netflix IP range> -> <has fast.com ClientHello> -> unthrottle. You don't need to do packet inspection on every packet, just ones that you might be interested in (e.g. Netflix IPs).

It's crazy to me that the many people who care about privacy and censorship in tech haven't pushed ECH (encrypted ClientHello) harder. It's such a gaping hole in web privacy that you can still passively snoop domain names sent in cleartext. It makes DoH/DoT almost pointless.

This isn't applicable to fast.com as it simply makes requests to Netflix's CDN from the frontend. Indistinguishable from regular Netflix traffic.

Good point, although you could still do logic like only activating the throttle after the customer visits netflix.com. You can't distinguish the CDN traffic, but you can still tell what website is being viewed.

Incidentally, my speeds on fast.com are always terrible (about 1/8 of what I get elsewhere), despite the fact that I'm fairly confident it is not being throttled. That's because the speed I see is >100 Mbps, which is like 4 Netflix UHD streams. Wouldn't be much point in throttling to that speed, you'd want 10 Mbps max, and less on wireless.

then you might miss embedded clients, right?

People drastically overestimate the security properties of TLS.

The correct mental model is that it’s good enough to convince 1990’s US internet users to type their credit card into a web page. (Where the downside of a breach is that you have to dispute some charges and change your CC#.)

If you need stronger security than that, then many, many caveats start to apply.

For instance, by default, anyone that can reliably man-in-the-middle port 80 on your website can get an acme certificate for your domain from a reputable certificate authority.

Is there more on this mitm cert spoofing somewhere I can read up on? Sounds intriguing to say the least

It probably isn't too different from other access methods: 1) Get access to port 80 for > 60 seconds. Point it at your temporary VM. 2) Use any cert authority, and for verification, choose a file-specific location verification (you can choose amongst DNS records, an email to admin@domain, or a specific file location on your site with many of them) 3) On your VM, Quickly paste the file-specific location into a django GET route. 4) Run the Django site on port 80. 5) The cert authority verifies you, and emails your account the cert, the website author being none the wiser. You can now use it later to fool future visitors for a deeper attack or email-related attacks.

This doesn't work unless the attacker happens to be in between your servers and the cert authority. The ISP that's in-between your laptop and the site can't pull this trick.

Also LE actually knows this attack is possible and mitigates it by validating the challenge from multiple sources so the attacker would need to be in the middle of all the LE validators and your servers.

https://portswigger.net/daily-swig/lets-encrypt-deploys-new-...

Yes that's true, but I was just talking about the scenario of having access to port 80 of a server DNS pointed at by some domain.

You might have access through editing a proxy rewrite rule, for example.

In the attack above you use your own SSL provider for a cert (LE not involved) and you overwrite the cert in a sense that existed before. You choose a provider that just validates with a file location. It's an attack which already requires compromise.

> The cert authority verifies you, and emails your account the cert, the website author being none the wiser.

It’s not hard to monitor cert transparency logs, which will show this new cert.

If by “people,” you mean “people who rely on HTTPS for security,” then it seems like you’re saying that every hosted software company is getting it wrong. HTTPS is the foundation of the “zero trust” security model as implemented by Google, MS365, Facebook, AWS, etc.

I think it’s more likely you are not considering the full picture of the TLS ecosystem, or are making arbitrary distinctions like “cert transparency logging isn’t actually part of HTTPS” or something.

Consider that Symantec basically did what you suggest (mis-issue some certs) and not only was it detected and mitigated, they lost their CA business entirely over it.

>For instance, by default, anyone that can reliably man-in-the-middle port 80 on your website can get an acme certificate for your domain from a reputable certificate authority.

You are leaving out a huge caveat here - exactly where the MITM is taking place matters a lot. In 99% cases this isn't possible unless the victim server network is effectively compromised.

Fast.com loads from nflxvideo.net for me. They have a /speedtest/ path.

Sure, they could apply some kind of advanced DPI assessment based on packets sizes over time and other crap, but if anyone complains it's much easier to just say "well, does speedtest.net work? What about Google's speedtest? Hmm, strange, must be fast.com being slow then!". If they just check for fast.com in the SNI header, I know I'd be loading the speedtest every time I want to watch Netflix.

ECH is slowly coming along, but it's still perfectly possible to detect these speed tests. It just takes much more time and effort to set up right.

I'm honestly surprised the article shows enabling a VPN actually working to fix the messed up network here. That would be a perfectly valid solution in my opinion; if I notice my ISP is messing with my network like this, there's no way I trust them enough not to use a VPN.

AT&T is well know for doing deep packet inspection so that they can properly throttle NetFlix. They are always at the top of the list of Net Neutrality violators that Netflix complains about.

I recently discovered that T-Mobile does this too, but they actually let you disable it on their site. Ostensibly, it's a feature for your benefit (somehow), and they're doing you a favor by enabling it by default. In reality, of course, it's for their own benefit, and they're banking on people not realizing it can be disabled. I suppose giving you the option lets them advertise things like "no throttling" and "4K streaming supported" while still reaping the benefits of throttling/lower-bitrate streaming.

>it's a feature for your benefit (somehow)

They don't count the "shaped/throttled" sites against your data plan limits, so I can see some people liking it.

They do this even with an unlimited plan.

T-Mobile gives me full speed when I set my APN to B2B/IPV4.

> Ostensibly, it's a feature for your benefit (somehow), and they're doing you a favor by enabling it by default.

Some throttling by default for video actually is sort of for your benefit, at least indirectly, although I don't know if carriers are throttling more than that amount.

A surprisingly large number of people nowadays watch movies on their phones or small tablets. On screens that size at the distances they typically hold them from their eyes 4K and usually 1080p won't offer any visible improvement over 720p. On some devices even 720p probably won't offer an improvement over 480p.

It is to everyone's benefit if other people on the network don't stream at a higher resolution than is necessary to reach the limit of what is visible. Since most people aren't going to change any settings from the default, defaulting to low speed streams maximizes benefits.

Speak for yourself. 1080p is noticeably worse if I'm paying attention; 720p is noticeably worse at a glance; and 480p (which is effectively T-Mobile's limit by default, even though they advertise my plan as supporting "4K streaming") looks like hot garbage.

Indeed that's the point of fast.com.... To measure the speed from Netflix servers

> Since fast.com is a Netflix ip

Technically the website fast.com resolves to an Akamai IP, but the speed test it loads uses a Netflix owned IP.

Reply on news.ycombinator.com