Comcast and Netflix now have a direct adjacency | Hacker News Reader

Comcast and Netflix now have a direct adjacency

gist.github.com

75 pointsbdb68 comments

It would be funny if netflix just looked for one of their hubs/datacenters and moved in next door on purpose.

ISPs are common carriers and must be regulated as such, because as soon as Comcast makes its own netflix-like service, you can forget getting netflix to stream smoothly.

I've done this before for large scraping projects. I find the datacenter the target website is hosted in, then get a dedicated server right next to it. I've never gotten better performance.

How large of a scraping project are we talking about in terms of throughput?

Largest was over 300TB. I talked with the owner beforehand and got access to the internal IP address so traffic wouldn't leave the datacenter (free of cost).

I offered to help them set up an API instead of scraping, but they decided scraping was easier in the short term.

Putting aside legal issues, you don't have any moral problems doing volumes of scraping content that is not yours?

Why should he? It's publicly available information.

Over time, I've learned to wget every web page and content archive I want to keep. The Internet forgets.

That's personal use, I have no problem with that. The above project sounds commercial in nature.

That seems pretty presumptuous...

In an earlier age, I ran everything through squid to consolidate browser caches. About five minutes after setting it up, I realised that pulling all the references in the log file and then indexing the lot with htdig would be tremendously useful when I was on the road without internet access.

I spent way too much time pruning stupid crap such as slashdot and started to learn this 'Bayesian classifier' thing.

Your idea is much better.

Certainly not! If I was re-selling the data, maybe. But I'm generally using it for statistics and data viz. I include the source of the data and I always obey robots.txt. Sometimes I'm even able to talk with the owners beforehand to get their ok.

(Don't downvote him, it's a valid question)

Now you'll have to tell us about the project...

300TB is quite a lot, even today.

dumb question, but how were you finding their datacenter?

traceroute is a good starting point. Sometimes I have to try a couple different datacenters until I hit <4ms ping time. Sometimes I just ask the datacenter is website x is hosted there.

The easiest way is usually to run a 'whois' on the IP of the server (not the domain).

There are only a handful of buildings where you need to be in order to just be a cross-connect or switch fabric away from nearly every network in the world.

Comcast has it's own netflix-like service. They call it something like Xfinity OnDemand. It's actually not terrible.

Except for the STB interface. At least the mobile app is decent.

Are you on X1?

Not available here. A Google Fiber truck was spotted in Atlanta, so I wouldn't be surprised if I can get that out in Winder before X1.

Streampix (http://www.comcast.com/streampix)

It's not like Netflix is fantastic by itself. It's serviceable, but the player is very rudimentary, and often has severe issues with things like subtitles.

If you're talking about streampix, read the disclaimers [1] and see why Netflix still wins (availability, selection, not tethered to one set-top-box, always on-demand, no random fees/taxes tacked on, doesn't require bundling to a subscription level). Until that changes, Netflix has nothing to worry about.

"Not available in all areas. Set-top box required to access On Demand on TV. Programming not available On Demand in all areas. Basic Service subscription required to receive other levels of service. Not all programming available in all areas. Equipment, installation, taxes and franchise fees extra. Pricing subject to change. Streampix included with the following tiers of service: Blast Plus, HD Preferred Plus XF Triple Play, HD Premier XF Triple Play and HD Complete XF Triple Play. Services and features subject to change..."

[1] http://www.comcast.com/streampix (see the legalese at bottom)

> as soon as Comcast makes its own netflix-like service, you can forget getting netflix to stream smoothly.

Comcast owns NBCUniversal and a 1/3 share of hulu as well as an "ondemand" service through the comcast cablebox and the internet, "soon" happened already. And in many places netflix on comcast already does not stream smoothly (almost unquestionably due to throttling of netflix traffic).

Comcast agreed to not throttle competing video services, at least until 2018, back when it acquired NBC (and so part of Hulu, which it had to step back from management from) a couple of years back.

So if you can make a case that they're "almost unquestionably" throttling Netflix, you could probably find some people very interested in that... but it sounds more like they were just saturated and are now taking steps to alleviate it.

Comcast already has a competing service: Cable TV

It's anti-climactic that all it was going to take was a compelling business case. Netflix made it easy with their peering initiative [1].

Now Comcast gets to count these bytes against their customers' quotas, and it costs them nearly nothing to deliver the traffic.

This reminds me of NNTP, but Netflix is still running their own hardware.

[1] Netflix's "Open Connect" https://signup.netflix.com/openconnect/guidelines

Most boxes like "Open Connect" pay to be colocated and connected that way. Netflix insists on free installation.

Not true. Big CDNs have been cutting deals like this for a long time. It's often in a network owner's financial interest to host a node for free, because it saves them so much upstream bandwidth which they would otherwise pay for.

Most CDN's pay money for this...

That's pretty hard to generalize. I worked at the biggest CDN for many years. They definitely get free traffic in many places. They had a whole team who's job was talking to network owners and saying "look, you paid for X terrabytes of traffic to us last month. We can cut that cost by 100 if you let us install servers inside your network that will cache the popular content locally."

It all depends on relative size. Bigger networks can demand money from smaller networks and/or CDNs. Networks of comparable size can profitably peer with each other without exchanging cash. Comcast may not do it for free, but the national ISP in a smallish developing country sure might.

It's a big dance, and the relationships are constantly changing. Managing it all in software to actually optimize cost and performance is a big part of the secret sauce for a CDN.

Yeah, it's a value proposition. The argument ISPs keep using is that Netflix traffic is costing them too much.

I don't think Netflix should have to engineer a solution to that problem AND pay the ISP for the privilege of saving them tons of money.

If the costs (4U of rack space, networking equipment, and network engineering around privacy/security) are more than the ISP would save, it's an easy decision.

Note that in this case, it does not appear to be the use of an appliance. If that were the case I think we'd see the Netflix content coming from a Comcast IP.

No; no ISP would ever run externally managed equipment inside their network. They would run it in a separate cage with a direct fiber connection to the gateway routers. Rule #1 of network security is that you never run someone else's equipment inside your network; you ALWAYS make them go through a gateway.

Also it's largely semantics where a device like OpenConnect is hosted. Netflix could host it at a datacenter across town with a direct fiber interconnect to the ISP and it would be effectively the same thing. This is how all the big CDNs do it; that way they host equipment once and connect to multiple local ISPs.

> No; no ISP would ever run externally managed equipment inside their network.

You haven't been paying attention to that Edward Snowden fella, have you?

Just stating the obvious here, but "Rule #1" obviously doesn't apply to service provider networks, who are specifically built so other people's equipment can be connected on almost every point. Customers on the access layer. Datacenters, colocation and these sorts of boxes on the distribution layer, and peers on the "gateway"/peering layers.

The second paragraph is wrong too, there are much closer relationships possible. They're much deeper than that today, and I would expect in the future to see CDNs much deeper in provider networks.

> This is how all the big CDNs do it

You have no idea how big CDNs do it. I've personally installed CDN gear into ISP racks. Sometimes you get an uplink into a router, other times you sit on a switch with other gear.

That "direct fiber interconnect" is called private peering and mostly used to fill cache boxes on the providers network.

Large ISPs sure as hell don't do this (maybe they used to; but not in the last 3 years). They have dedicated cages in their datacenters for external gear that sit at the edge of their network.

Regardless, the word on the Netflix-Comcast situation is that Netflix is indeed hosting the hardware at 3rd party datacenters with a dedicated connection to Comcast. Whether you call it an interconnect or private peering is just semantics; it's a pretty common practice in the industry and technologically, it's no different than having a 10gig fiber link within a datacenter.

I spent a year in 2007-2008 managing equipment inside Comcast's network while working for a very much not-Comcast company. Even had a Comcast VPN assigned specifically to me complete with RSA fob shipped to me by Comcast. You haven't got the first clue what ISPs do, nor how networks work.

I wouldn't be so sure there's no money changing hands. One of the sides caved; and it was probably Netflix. The amount of money the ISPs were asking for from Netflix was not outrageous; the ISPs were more worried about the precedent that providing free interconnects might set. Give one to Netflix for free, and Apple, Google, Amazon and Microsoft all think they're entitled to one too.

Strategically, Netflix was holding a far weaker hand because the ISPs had no reason to give in since their brand perception was already so bad. I mean really, is it possible to hate Comcast more than most people already do?

> is it possible to hate Comcast more than most people already do?

Yes. Right now the average person only hates Comcast because its a monthly bill, it increases frequently, and there is not much competition, they don't yet hate Comcast for the technical reasons "internet" people do.

But if Comcast starts making Netflix, YouTube, etc not work correctly and starts affecting the average American's daily life it will become a political issue. Congressmen will start seriously discussing turning the internet into a utility.

> Congressmen will start seriously discussing turning the internet into a utility.

I loled. Congress has a lot more important problems on its plate than this (social security, immigration, health care, etc.) On this issue they're just going to do what ever their friendly telecom lobbyist tells them to do.

> Congressmen will start seriously discussing turning the internet into a utility.

They have already done this, it should be opposed. You don't want people who can't balance a checkbook and think that the internet is a "series of tubes" deciding how you get internet access. The PUC/FCC rules and tarrifs are already a byzantine nightmare.

I always thought that Netflix served everything from AWS. How can they they have their own AS number and peering locations?

Yeah, you were wrong :)

Netflix's website is hosted from AWS they don't stream moves from it. All the video data comes from a CDN. In the past they have used pretty much all of the major CDNs before creating their own.

I've noticed that the 11greatoaks.ca.ibone.comcast.net router(s) (Equinix SV1) are typically the ones that fail / have large latency issues. Hopefully if this has happened then they've increased overall capacity through this bottleneck.

You're seeing this because that's where Comcast primarily buys transit and peers with other networks; those edges are where the congested ports are. They generally have plenty of capacity between SV1 and their CMTSes (even if they have to take you from Oakland to Sac-town to get from SF to San Jose).

Comcast has been horribly congested in the SFBA for months.

I believe they've been looking at offloading that congestion and frankly moving Netflix is a no brainer.

Perhaps it's Netflix Open Connect? (http://oc.nflxvideo.net/docs/OpenConnect-Deployment-Guide.pd..., https://signup.netflix.com/openconnect)

I can confirm this:

198.45.63.0/24 *[BGP/170] 2d 05:02:06, MED 150, localpref 100, from 68.86.80.82

AS path: 7922 2906 I

I will admit that I am probably making a pretty sweeping assumption here... but I'm assuming Netflix previously had the same access as everyone else on the Internet, from Comcast - and now, they do not. They have BETTER access.

To me, this is disturbing. Surely there is some financial incentive for Comcast to do this.

It seems to me that this is exactly the kind of thing that the whole "net neutrality" issue is trying to prevent (i.e. back office deals that give one content provider better access over others)

Or... am I just missing lots of things? (wouldn't be the first time!) :)

I did notice last night that my stream of House of Cards looked way better than it has in the past month (when it started to go bad). It was HD level the whole hour, while previously it would only go HD for about 10 minutes total randomly through the episode.

I'm a Comcast user in San Mateo, CA.

That is probably my doing :) Comcast had some internal congestion issues that we helped illustrate.

Can you tell me why Netflix is so terrible on Uverse 20Meg in San Jose?

Blurry mess.

Speedtest is always 13Meg+

Its the same reason Comcast was so bad, except you have ATT. Different companies, same problem.

Switch to Sonic or something and that problem will go away.

and what's the problem exactly?

Endemic congestion.

Isn't it possible that the traffic could just be going over an MPLS backbone? If that's the case, then there could potentially be more hops that aren't seen.

Thats not relevant to traffic interchange...

A short route doesn't necessarily mean good bandwidth

It does reduce the buck-passing. See the current Verizon-Netflix squabble using Cogent as a proxy.

As a FIOS customer, it really is delightful. Especially when you're trying to get actual work done from your business FIOS connection to your rack in a colocation facility, not watch videos.

Only after calling and complaining for a week did they change the route, and magically, the latency and 50%+ packet loss went away.

How are you concluding that from the tracert?

Using the Equinix IX https://signup.netflix.com/openconnect/faq

I don't think so. It's multiple direct 10GE ports. Given the traffic volume the two networks exchange, there's no way it would be economical to move these bits over a public peering exchange (which Comcast doesn't participate in in the first place).

Even so, if it were in fact going via the IX, you'd see Netflix's IP from the exchange (206.223.116.133) as hop 8 in the traceroute.

Am I understanding this right? If I am an ISP, I can request a Netflix appliance and serve customer requests for netflix content from this appliance directly?

Basically... If you're moving 5Gbps+, contact Netflix.

Meh. So your ISP joined the Netflix open connect CDN to make your Netflix experience even slower.

Have fun blocking that address and other known Netflix cdn

Reply on news.ycombinator.com