Building fast.com
techblog.netflix.com
techblog.netflix.com
I strongly suspect some ISPs already do that with speedtest.net.
https://github.com/sivel/speedtest-cli
If you are going to setup a cronjob/service for it, I suggest using the --simple flag.
speedtest-cli has a lot to offer as a tool
It provides cleaner information quicker. Yeah you could certainly setup a cron job using wget and have it do everything you want it too down to the bits it changes in memory... That just offers a quicker way to do all you may want too with that information.
The domain is exposed over various ways, yes, but that's the point of the idea to connect to fast.com regularly to trick theoretical shaping based on it.
Not bad.
EDIT: And they secured a fantastic domain.
Perhaps it's encrypted, or a Netflix promo piece.
Wouldn't there be copyright issues to use their actual streaming licences content?
The interesting traffic is not to fast.com.
It would be harder (though not impossible) if there is an actual capacity problem, and the bottleneck is outside of the ISP's network. For example insufficient peering capacity, which seems to be where most of the Netflix / ISP friction is.
That presumes they aren't fully emulating streaming traffic to test speed. Not only is this the correct thing to do from a video streaming company that wants to test connection speed for their services (the quantity and size of packets can affect the delivery through different mediums and devices), but it also makes it obvious when your ISP is quietly throttling your video streaming traffic.
Since it's a win-win in this respect, I would be surprised if this isn't exactly what they are doing.
It's not really a presumption, it's the actual facts on the ground. The actual benchmark file is served over HTTPS from, but there are still multiple simple ways to distinguish the current speedtest testcase from normal streaming.
Do you mind explaining the details you are aware of that lead you to think they are not emulating streaming traffic in at least some portion of their test (since they are testing multiple sessions and types of traffic)? I may have missed a portion of this article or some other source that outlined this.
It would fairly trivial to send sample content of actual video content for the test, and emulate a streaming client on the tester side, so I'm not sure how an ISP would distinguish traffic in that situation.
I agree, that in this case, where a separate domain was used, ISPs can use that to detect a speed test is likely. That said, I would like to note that every solution an ISP has to implement is much more expensive to any mitigation put forth by Netflix, as long as they want to hide what they are doing. If they aren't hiding it, well that's one of the reasons netflix is doing this, to inform customers what they are actually getting from their ISP, and let them make choice whether they like the current product they are using from their ISP (if they have a credible choice).
Ultimately if they're playing cat to Netflix's mouse how much does each mitigation strategy cost them relative to Netflix? I wonder if it's more cost effective to process that much traffic through shapers than it is to increase bandwidth between peers or allow Netflix to drop their boxes within their networks.
Like looking out for DNS queries for fast.com then handling traffic from that client differently for a period (such as the A record's TTL).
Then, I checked how much traffic speedtest.net is getting on SimilarWeb: wow, 150m visits per month, I didn't know that speed testing can be so popular.
Conclusion: Yes, it was right to buy this domain and if fast.com gets a large part of speedtest.net's traffic in the long term, Netflix has a neat free Marketing channel.
So I'm not enthusiastic at all.
That's nontrivially hard compared to the download portion, where they can use their existing CDN.
Sure -- they didn't build out a fully featured internet connection monitor. But what they have built is a pretty good tool, especially for diagnosing Netflix or last-mile performance.
Apparently they have slow.com as well.
Speedtest 12.6 Mbps
Fast.com 13.0 Mbps
Speedtest 40.0 Mbps
Fast.com 73.0 Mbps> So you are really testing the bandwidth of the last mile to your house and the ISP's internal network, which is not representative of your real network bandwidth to the rest of the Internet.
There's no such thing. If you were downloading a file from a friends computer and they have 2mbps upload, your connection speed will be 2mbps. How would your ISP have anything to do with that?
Fast.com just measures your connection to Netflix, which is valuable if that's a service you use a lot.
Yes there is. The Internet is not an abstract cloud with unlimited bandwidth everywhere. Your ISP runs their own network, peered with other bandwidth providers or other ISPs. You may have a 50 Mbps downlink from your ISP, but it doesn't mean the real-world usable bandwidth to all hosts on the Internet will be 50 Mbps, because your packets will be going through these peering links when they leave your ISP's network.
"How would your ISP have anything to do with that?"
Well, simply put some ISPs have terrible peering and/or intentionally throttle. So speedtest.net will show you awesome numbers, but in practice anything you download will be at much lower speeds. This matters, this is precisely why fast.com was built, and this is why the GGP is seeing different bandwidths when testing with speedtest.net vs. fast.com.
Perhaps it mitigates against proxies stripping the headers, or it simplifies caching.
One of many reasons to use HTTPS: prevent tampering with your requests and responses.
I find it particularly telling that switching to HTTPS caused many streaming errors to mysteriously vanish, and I suspect ISP / proxy tampering to be part of the cause. I'd love to know more about what their engineers saw though.
Yes, see http://stackoverflow.com/questions/15561508/xmlhttprequest-2...
> Can you set them from random video player SDKs?
Good point, probably not.
Edit: not sure what format they are using, probably several depending on the client... But one url per chunk makes sense for multiple delivery formats as well.
The formats that send time ranges instead of bytes ranges are the worst. In that case the server/cdn has to retrieve & read the manifest, decode time indexes to fragments/chunks/ranges, and then return the appropriate byte stream to the player. This also means you cant cache based on the URI plus query string!
Source: ive worked at both ISPs and CDNs supporting streaming content
fast | cat
860 Mbps $ fast --help
Usage
$ fast
$ fast > file
Example
$ fast | cat
90 Mbps mkfifo /tmp/fast.com.test.fifo ;token=$(curl -s https://fast.com/app-ed402d.js|egrep -om1 'token:"[^"]+'|cut -f2 -d'"'); curl -s "https://api.fast.com/netflix/speedtest?https=true&token=$token" |egrep -o 'https[^"]+'|while read url; do first=${url/speedtest/speedtest\/range\/0-2048}; next=${url/speedtest/speedtest\/range\/0-26214400};(curl -s -H 'Referer: https://fast.com/' -H 'Origin: https://fast.com' "$first" > /tmp/fast.com.test.fifo; for i in {1..10}; do curl -s -H 'Referer: https://fast.com/' -H 'Origin: https://fast.com' "$next">>/tmp/fast.com.test.fifo; done)& done & pv /tmp/fast.com.test.fifo > /dev/null; rm /tmp/fast.com.test.fifo2. I think tests like this that rely on a built-out network and well-placed CDNs and such are probably junk. A more useful test would be against a handful of download targets that aren't optimized, and then average the results. This is particularly true for Netflix. Case in point: fast.com says I download at 78Mbps, but speedtest.net has me down around 57. I'm only paying for 30.
Depends on the goal - here Netflix wants to see how fast you can stream from their "built-out network and well placed CDNs" so testing those is completely relevant. For their purposes, testing a bunch of unoptimized and unrelated services would be "junk".
Absolutely right; but also in this case, Netflix' and the average downloader's needs align, given that the bulk (by share of bytes) of legal speed-sensitive downstream traffic is video.
I'm firmly untrusting of Comcast's results on speedtest.. If Comcast is intentionally throttling, it's in their interest to not throttle Speedtest. However if we can monitor real data.. that actually seems useful.
I understand that would likely require additional resources from video/etc providers, and is likely an unrealistic request from video/etc providers.. but i definitely don't need more test sites for Comcast to manipulate.
Disclaimer: I don't know the subject domain, so apologies if i'm insanely off base.
Granted we do not have many results in the US (maybe ballpark 200k+ results in last 6 months), still you should be able to find some results in your area.
Most election/political polls use far less data and very big decisions get made because of them.
Maybe netflix messed up their HSTS settings at some point in the past?
It's weird because in its settings, there is nothing in the "force https website" list, and the "forbid http" section is set to never.
If you get a solution, please let me know.
The best solution is to go into Noscript options > advanced > HTTPS and add .netflix.com to "Never force secure".
The other way is to go into about:config and set noscript.httpsDefWhitelist to false. But that's less secure.
My ISP hasn't even officially announced any IPv6 support, I discovered more or less by accident that if I enabled 6rd things "just worked" so the fact that it might not go at the full speed of my subscription isn't something I'm super worried about.
I've actually got a similar issue with Google. While using IPv4 I don't even leave my ISP's network as they have those Google Global Cache boxes at the edge and terminate my connection there but over IPv6 I do go all the way out over the internet to Google.
Yes, that's the case since, as I stated, I got IPv6 connectivity through 6rd:
> It is derived from 6to4, a preexisting mechanism to transfer IPv6 packets over the IPv4 network
The net result of applying their capital, engineering talent and existing infrastructure here: a positive recruiting impact for engineers, a positive branding impact for consumers visiting fast.com, and Netflix obtains information about internet speeds for IPs that are not currently Netflix customers, information that many businesses would find useful for things like determining the market for high bandwidth applications.
I get an insignificant discrepancy: 91 on Fast and 91.99 on Speedtest.
Netflix pushes their CDN nodes to the limit, 60-70Gbps per box for video. Speedtests don't make them money, so why provision capacity for it?
It seems Netflix does shove info to you as fast as they can from fast.com
I have 1Gbps CondoInternet/WaveG at home, right now at 11PM PST I get ~400Mbps from Fast.com and 800Mbps+ from DSLReports and the Comcast/Xfinity speedtest.
Good work by Netflix.
I think this is an interesting idea, but I'm not convinced it'll actually make a difference. From the consumer side, there's nothing to say that the slowest link isn't Netflix's CDN when a discrepancy occurs between Fast and Speedtest.
I was receiving atrocious download speeds for movies from iTunes (18+ hours for 4 GB), and also saw slow speeds from speakeasy.net/speedtest. The comcast rep just pointed me at Speedtest and said "it's not a problem with your connection, it's fast! Contact Apple." He dismissed the "other" speed testing results. ISPs can continue to throttle directly or by refusing to upgrade peering connections. fast.com providing a number to go along with bad Netflix streaming quality doesn't improve the customer's life.
What I really want is something that empowers me to be heard in my request for the $75 worth of Internet that I pay for every month. I had high hopes that the SamKnows program might yield something, until I received a docsis 3 cable modem in the mail from Comcast with a note: "since you're a participant in SamKnows monitoring, we sent you this upgraded modem so you can take advantage of better speeds". My first thought was "what about all of your customers who aren't participating in SamKnows? When do they receive their 'upgrades'?" Do we really need the government monitoring everyone's broadband performance to ensure that companies are delivering?
Their response? "We use SpeedTest.net as standard, it must be a problem with the servers that you are using".
Ever since Netflix became available in our country we have been hurting BW wise. We don't throttle BW for any sites. Does anyone here have any experience with open connect (Netflix)?
https://openconnect.netflix.com/en/requirements-for-deployin...
I'm willing to bet additional factors that went into that consideration were:
- Netflix gets massive download traffic on the cheap due to their scale. Upload, not so much
- This tool is a clever way of preventing providers from throttling Netflix servers without also throttling a (hopfully soon to be) popular speedtest. Uploading isn't something they need their primary customers to be able to do.
You can easily determine how much of a problem it is in your case by calculating your bandwidth delay product: https://en.wikipedia.org/wiki/Bandwidth-delay_product
For many activities on the internet a 1,000ms latency would be intolerable, but for one-way video streaming it's unnoticeable to the end user. I don't really care if a packet containing video takes 50ms or 1000ms to get from the CDN to my TV, as long as it gets there and is reasonably consistent in it's ability to do so.
For this use case, if my machine can get 20Mb/s on the test, latency will just not affect me streaming a 10Mb/s video. Latency may be lowering my test results, but that's the point here - testing the effective throughput.
The good news is that many of the smaller providers (where they exist - maybe your LTE provider is one?) are very excited about adding something like this to their network. Where the incumbents would rather just hide problems from you to keep you locked in, the smaller providers are very motivated to keep you on their network. They're often overworked and resource strapped - so a system that will help them narrow down the source of network problems and reduce truck rolls is usually very exciting to them.
"Could not reach our servers to perform the test. You may not be connected to the internet"
P.S.: It also bugs me that it says "the internet" and not "the Internet". :)
"The changes reflect a growing trend toward lowercasing both words, which have become generic terms"
http://www.poynter.org/2016/ap-style-change-alert-dont-capit...
In general, I've been quite happy with Cox. The actual network service is great. Customer service and pricing are so-so (I'm paying $60/mo. for the above connection; customer service is very hit-or-miss depending on which representative I get, but I haven't been outright lied to).
It provides the ability to do lots of troubleshooting in a private network.
Free/dockerized: https://hub.docker.com/r/dperson/smokeping/
But tests shows 5mbits for me and all other 30+ (real). So not really a great help till now.