Netflix Agrees to Pay Comcast to End Web Traffic Jam
online.wsj.com
online.wsj.com
For $70 a month from Comcast you can get Amazon, Google, and Facebook. For only $10 more a month you can add Netflix and Spotify. Want Pandora? Sorry, you'll have to switch to AT&T for that.
The broadband providers are used to this type of packaged service given their history as cable television providers. It's how they make their money beyond providing simple connectivity. Their future expansion and profits are linked to their ability to create exclusivity and extract revenue based on targeted demand.
I don't see any other way around this. Absent regulations prohibiting packet discrimination, I'm not sure I would do any different if I were in Comcast's shoes.
Vote with your feet.
Because it's either Comcast or a DSL provider that won't give me a straight answer as to whether or not they service my address. And this is in Boulder.
Imagine if that fiber were used as a community network! Unfortunately, the town is not planning to do this -- they've excluded the fiber from their plans to take over the Xcel infrastructure.
Do a lot of people in the US only have access to one or two ISP's?? If so why? I'm in a not particularly urban part of Ireland and I can choose between at least 5 ISP's, decent speeds, and a good variety of prices.
Many people in the US have quite a bit of choices: Cable, DSL, Cell+Internet, Wireless. Usually those that live in large urban areas. But the service and quality of these ISP's varies greatly, which Comcast--in my area, being top.
Those Americans that live in rural areas, may not even have access to high speed internet, or always on internet--see the Xbox 1 fiasco for more details.
This needs to be emphasized more. We rarely get to choose between equally cost-effective ISP options. The typical choice is one cable company, one phone company, and some kind of wireless service (possibly satellite). That is not competition, any more than heavy truck sales compete with mo-ped sales.
The BT Group was also forcibly split into separate operating companies, particularly OpenReach which owns the local loops and sells services to all ISPs including BT Retail.
I live in a middle/upper-middle class suburban area in upstate New York.
If you live in a sparsely populated area the excuse is "well, it is expensive to wire you up because of distances". If you live in a densely populated area the excuse is "well, it costs a lot of money to put new infrastructure into a densely populated area".
In both cases, lots of other countries are managing to do both of those at far lower cost to the consumer than occurs here in the US. Ultimately the ISPs here in the US are just greedy, duopolistic fucks.
The only factor that determines the quality of your ISP options is having a competitive player in the market.
Most places are wired with one phone line, and one cable line. That phone line is generally owned by one specific phone company, and that cable line is generally owned by one specific cable provider.
So those are the choices of ISP: you can choose cable service or dsl.
For a few years the last mile over phone had to be shared with other IP providers, so you could choose between a) one cable company, or b) a few IP over DSL ISPs, but that ended sometime back.
There is also satellite providers, but they usually, I believe make a deal with your phone company, wireless, which is expensive, and fiber, which to my understanding is still just an upgrade from the over cable company.
How is the last mile provided in Ireland? And what are your choices of ISPs?
That seems to be where things differ. My phone line is owned by BT but I can still order internet through a different company. I pay that company an extra £10-15 per month for line rental (so I guess BT, even though they own the line chooses/or has to rent it to others).
I live in a suburb of a medium sized city. At my house, I have 3 choices for wired Internet access: Knology/WOW, AT&T DSL, AT&T Uverse. DSL is right out because it's too slow, so really it's a choice between Cable (Knology) and AT&T's Uverse (which is FTTN or FTTP depending on various factors).
We moved here a couple years ago, two miles away from our old house. Same city, same suburb, just 2 miles difference. At our old house, we didn't have Uverse as an option - it was just slow DSL through the phone company or a single cable provider. If I had lived three streets over (less than 100 yards), I would have had a choice between Comcast or Knology cable, and DSL.
In other areas of the suburb that are further out, you start to see Mediacom (another cable provider). There are probably some areas that Comcast and Mediacom overlap in.
I have a coworker that lives in a rural area, and all he has is Windstream DSL. Cable doesn't reach out that far.
These same areas are often also serviced by wireless providers, either cellular, satellite or some other kind of fixed wireless.
Note that while DSL tends to be available in every market, it may not actually be available at your home due to distance from the CO. Or it may be available but you're so far away that the performance would not be worth what it cost.
So the problem is that there's a clear hierarchy of what's good in any area. Where I live it's cable > DSL >> wireless/satellite. Compound this with the pricing structure: where I live it's wireless/satellite >> cable == dsl, with cable having some speed tiers that DSL can't match. At the end of the day, for any consumer trying to make a decision on value and performance, there's often really only one option.
I'm going to assert that this is how it looks for the majority of the geographical US. I'm not sure about the larger metro areas.
In the US, you generally have a choice between one (two if you're lucky) cable companies and DSL.
Unfortunately, the min/max slider forces hard-coded values during page load, so I can't link to specific graphs. I suggest checking out "served areas"/min:1/max:1 and "unserved areas"/min:2/max:whatever.
Those maps suggest the "1 ISP" set (or any set, really) is fairly uniformly distributed across the nation. In my opinion, it's not quite as biased towards the urban/rural divide I expected.
http://www.broadbandmap.gov/summarize/nationwide
#ISP %USA
0 3.3%
1 8.9%
2 32.0%
3 35.6%
4 14.3%
5 4.1%
6 1.3%
7 0.4%
8+ 0.2%
Regardless, 8.9% of the nation is still a LOT of people, and "2 or 3 providers across the nation" isn't really a thriving economy full of healthy competition.
The interesting data points, I think, are the high end. A 4-ISPs isn't THAT different from 1-ISP (8.9% vs 14.3%), and those numbers drop off fast in the 5+ range.
Oh, and yes, I'm excluding radio, in any form ("G[0-9]", various satellite/asymmetric services, etc). Those are a different product entirely, and any similarity in feature set is only superficial. You can't play games or any other low-latency task over most wireless, and VoIP doesn't like the "several second" latencies of satellite, for example.
HBO charges Comcast a flat rate to carry their content, because HBO has great content.
Comcast then bundles HBO in various packages designed to bring a profit, knowing that people just want HBO, and are willing to pay extra for a bundle in order to include it.
In this case, Netflix is paying Comcast for the rights to use their End of Mile network.
With Comcast I get ~3MB/s internet. DSL and Wireless broadband aren't even close to those speeds (in my area). So if I'm forced to pay a premium or switch, I have to "give up" significant performance.
PLEASE GOD GIVE ME GOOGLE FIBER.
Consumers are going to get screwed on the front & back end of this.
Keep in mind that this is New York City. You'll have lesser options in many other places in the US.
We have two FTTH providers (AT&T, Grande). Grande offers 1Gbps now, AT&T is 300Mbps now, 1Gbps this summer. Time Warner offers 300Mbps now (DOCSIS).
Google fiber is being deployed this year (1Gbps).
For my zip code they list AT&T UVerse, AT&T DSL, Cox and Time Warner. The reality of my actual address is that my only options are Time Warner and AT&T DSL (old school fully copper DSL, not UVerse). I'm positive these are my only wired options because I have researched this extensively (including actually calling the companies, not just relying on bad Internet data) due to being unhappy with my two options, and I continue to look into it about once every 6 months but nothing ever changes. Both AT&T UVerse and Verizon FiOS seem to have gone into deep slumber mode when it comes to service expansion.
The really sad part is that I've been using AT&T DSL for the past two years because while it is significantly slower than Time Warner cable in the bandwidth department, it has been far more reliable. Cost-wise both are a complete rip-off already, and if the government doesn't step in on the ISPs soon, I expect that the cost issue is going to get a lot worse, not better.
Also the information blurb at the bottom says I have a fiber provider, but none are listed.
Finally, it's missing some local providers which I am positive exist; our local power/water provider also provides Ethernet to local business and some residential areas, but it's not listed.
Netflix is network heavy and competes with Comcast offerings using Comcast's own network. That was the key motivator here.
Even if Comcast has a monopoly in select areas, it cannot do price differentiation in just those select areas. IANAL, but that sounds illegal.
Funny, back when I was a Comcast customer, I do not remember paying for "Comcast Network Access." I remember paying for "Internet Access;" sure, I went over Comcast's network, but I (maybe naively) thought that such distinctions were irrelevant on the Internet (that was kind of the point, if memory serves).
No more silly than pretending the the abstraction of "files" and "directories" is reality. No more silly than pretending that your computer is running a Python program, when in reality there are sequences of machine language instructions being run. Computers are useful because of abstractions.
"There is no Internet, just a bunch of private networks connected together"
You might have missed the definition of "an internet:"
https://en.wikipedia.org/wiki/Internetwork
There is, in fact, an Internet that we use every day. The overwhelming success of the Internet is due to the abstraction it presents, which is best-effort routing of packets from one host to another -- without regard for what networks the hosts are connected to, what arrangements govern their connections to those networks, and perhaps most important of all, what applications those hosts are running. On the scale of positive improvements to human communication, the Internet is up there with the printing press and written language.
"Now that real money is in play, the fiction of the internet is crumbling and the physical nature is becoming exposed."
No, what we are seeing is the impact of years of systematic attacks on the Internet by companies that are desperate to protect obsolete business models. Comcast is worried about the imminent demise of cable TV and all the power and profit that came with it. AT&T is worried about the demise of long distance fees, roaming charges, SMS charges, and all the power and profit that came with controlling the phone system. The recording industry and movie industry worked hard to leave the Internet divided into "consumers" and "services," even though the system itself was designed to be far more general.
In short, had we told these companies to adapt or die rather than tolerating their efforts to roll back the clock, we would not be in the situation we are in today. This has nothing to do with leaky abstractions, physical limitations, or the design of the Internet. It is just a power grab, a systematic effort to destroy a good thing that threatened the entrenched players.
Its a useful abstraction, but as physics causes CPUs to hit a performance wall on single threaded code, then you start banging up against the GIL and the failure of the abstraction to deal with parallelism. And when that happens you can't bury your head in the abstraction of a Python program. I'm not saying that abstractions aren't useful, but at the end of the day you have to concede that they are merely abstractions. When the abstraction becomes disconnected from the reality, reality wins.
> In short, had we told these companies to adapt or die rather than tolerating their efforts to roll back the clock, we would not be in the situation we are in today.
Adapt to what? Abstractions must adapt to reality. Reality does not have to adapt to abstractions. You can of course invoke the government, to regulate the reality into conforming to the abstraction, to the benefit of certain companies and the detriment of others. But history has not been kind to regulatory regimes of that nature. The deep irony of your position is that its the same thinking that led to the current mess with cable. In the 1980s, cable was the future. People invoked the potential to distribute educational material and public access material to the masses, and decided that such an important technology should be regulated. So they created the cable monopolies, under the conditions that they have cheap basic access rates ($13/month in Phila), carry public access or educational programming, and build out even to poor and rural communities. And it was an utter disaster in retrospect.
Further, assuming franchise agreements exist and were necessary, consumers and municipalities should expect better from their franchisees than abuse of monopoly.
Comcast worries about the FCC and Congress, not its competitors.
ISPs have the option of making content providers(Netflix), or content consumers(you and me) pay more. They're choosing content providers because they don't want to make net neutrality more salient for their customers.
It would seem that worry extends only so far. In this case, if what you're saying is accurate, Comcast intends to take (i.e., extract rent), and it just doesn't want to get noticed while it takes. Fear of the FCC and potential consumer backlash do not extend so far as to help it to reconsider its basic strategy of expanding rents, even as its merger with Time Warner is under review. There's a certain boldness to Comcast's move in working out this deal with Netflix.
Netflix isn't just going to absorb those payments, they'll have to pass them along to customers. Customers will want some explanation for why their bill is going up . And if Netflix's per-user costs become substantially different at different ISPs, they'll have to start thinking about ISP-specific surcharges...
At least once a month for the past couple of years, Comcast sends me some junk mail begging for an "Account Review," in which they would add triple-play service to my business Internet account in exchange for a discount on all three. Those letters go in the trash on general principles.
Peering is a practice where two sides agree to maintain their side of the infrastructure and not charge for metered bandwidth.
Comcast will also likely work with Netflix to estimate traffic growth and make sure their infrastructure scales before they have problems.
Again, we don't know the details of the deal. Assuming net neutrality was not damaged is also speculation. We need more information to know what is going on.
In old world there were a dozen of "tier1" providers who would all peer with each other more or less settlement free. It is conceivable that one of these providers (I think it was Cogent in this case) was offering "low rates" to take Netflix traffic and deliver that traffic to Comcast over their settlement free peering. As traffic flow between Cogent and Comcast was increasing, Comcast refused to upgrade the interconnect to accommodate the flow. Then: links get hot, traffic gets dropped, customers complain, Netflix+Cogent asks Comcast to behave, Comcast offers to unblock the pipes via direct Netflix<->Comcast peering for a "fee".
Now I don't know whether that "low rate" that Cogent was charging Netflix was lower than a "fee" that Comcast is charging. It might be that you are right and the fee overall reduced Netflix costs. What I do know however, is that Comcast is a monopoly and if unregulated, they can start extracting higher and higher rent from Netflix (or any other content provider for that sake).
Assuming it wasn't harmed is also speculation and we should definitely be concerned about the deal and what it means for net neutrality.
Generally speaking settlement free peering requires that the arrangement be mutually beneficial. If the traffic exchange becomes inequitable then the side that disproportionately benefits would need to pay to cover it.
In this case Cogent was sending vastly more traffic towards Comcast than they were receiving (because of Netflix) so they were the misbehaving party.
I'll admit my knowledge of peering is minimal, but consumer ISPs have always been something of an anomaly if memory serves. The traffic flow is highly one sided, but peering with consumer ISPs is obviously necessary if you want to actually reach consumers.
For example, they are the 30th largest provider globally going by the CADIA rankings (http://as-rank.caida.org/?mode0=as-info&mode1=as-table&as=79...) and have large traffic sources as customers. From the CADIA list this includes Peer1, Dreamhost, SoftLayer, Hivelocity, and Liquid Web.
As a eyeball ISP I occasionally get offered cheap transit from hosters looking to balance their traffic ratios. From the CADIA list it seems like Comcast may be doing the opposite.
This is one of the underreported issues in the whole net neutrality debate. In the late '90s I worked in the WAN engineering group doing capacity planning at a big regional CLEC, which actually was a Tier 1 backbone provider. It was widely assumed in the industry that the flat rate pricing model -- pay $X for Y "guaranteed" bandwidth -- was living on borrowed time and would be replaced by actual usage-based pricing, paying for packets like you pay for gallons of water. The flat rate model only worked because we generally could keep available capacity well ahead of demand. We could do that because we were adding customers like crazy, and customers sucking down high bandwidth were outliers. If those prerequisites ever changed we'd be screwed.
Well, here we are in 2014 and ISPs like Comcast are not adding customers like crazy anymore, because most of the people who want high-speed data from them already have it. But the customers they do have are, on a per capita basis, using much more bandwidth now than they were five years ago. And this is only going to get worse. The nightmare scenario is basically upon us: capacity requirements are climbing faster than ever, but those requirements are no longer being driven by customer acquisition. Which means either the ISPs eat the cost of upgrading their infrastructure (ha!) -- or they raise rates.
But wait, doesn't Netflix pay for all their bandwidth already? Well, probably not: unless industry practices have changed drastically, even those backbone trunks are actually oversubscribed. If you pay for (say) a DS3 line, there's an implicit assumption that you are not going to be pumping 44.736 Mbit/s through it 24/7. Netflix breaks that assumption.
I don't know quite what the solution here is, and I certainly don't want the kind of dystopian "the ISP only gives you access to web sites that you pay for" future the sharpest critics paint. But I don't see this as a step toward that dystopian future. What I see it as is the ISPs trying to figure out how to work usage into pricing models without saying "screw it" and actually adopting a usage-based pricing model.
Whoa, whoa, whoa. This is exactly the assumption commercial customers paying for unmetered links are making, otherwise they wouldn't be paying for unmetered links. Not being oversubscribed is supposed to the entire point. If ISPs' business models depend on them being able to break their promises to customers paying for continuously saturated links that's their problem, but they're still on the hook for what they're being paid to provide.
I personally think the resistance to usage-based pricing in the tech community is really weird, although I mostly see it on the consumer end. Hard caps are bad, and caps that only apply to some services are really bad, but charging more if you use more is completely reasonable.
(Not to mention that there are no inherent variable costs to transporting bits. Water needs to be purified, electricity is obviously usage priced - but you can be pushing bits forever once you have the hardware setup.)
With usage-based billing, the amount will float up and down and it's hard to understand how the amount is calculated. Usage-based telephone bills are broken down by calls, which you can probably remember.("What's this 46 minute call to 555-1234? Oh yeah, I called Mom last week.")
With IP traffic, how do you itemize the charges? Temporally? I used to get cell phone bills that included pages and pages of stuff like "Jan 27, 4pm, 239789987345 bytes". It was completely useless.
How about reverse DNS? That has other problems. • You use BitTorrent? Here's 247 hosts with incomprehensible names on your bill. • You use a VPN? A whole bunch of traffic is lumped together. • "Honey, what's all this traffic to nastyporn.com?" • It puts pressure on web sites to optimize their deployment strategy for billing clarity
The "correct" way is probably to log everything, then provide analytics software. But I can't imagine my Mom generating reports just to understand her internet bill.
Economically, sure, usage-based billing is a great idea. Alignment of incentives and all that. But practically, it's a pain for everybody involved.
My electricity bill shows only X kWh per month. There is no "itemization" of usage. However I can purchase watt meters that can monitor power anywhere (including the main grid connection) if I decide the bill is too high.
I can definitely imagine a similar service sitting on the router or provided by the provider that does bandwidth accounting to deliver a picture of what is going on. I'm sure there would be many third party device manufacturers willing to do this traffic accounting as well.
I dunno.. being profitable enough to buy time-warner cable for billions doesn't sound like a nightmare scenario from their POV. It sounds like they'd rather spend the money becoming a monopoly than improve their infrastructure.
If it is really expensive to accept Netflix's offer of free hardware and free bandwidth then the additional cost should be passed to the customer. The problem is that it's not actually more expensive and Comcast knows their customers can't choose someone else so it's a much better strategy to squeeze the content providers.
Netflix could have held out, and made Comcast out to be the bad guy, or could have leaned on Cogent to get the circuits installed - but, in this case, they folded and just decided to pay Comcast. This will probably set a very bad precedent for them when it comes to renew those agreements in 5-10 years, or however long they will be. Maybe they are hoping for some competition to enter the US environment by then.
How would that work? Comcast customers by and large can't switch to another broadband provider and Netflix customers can certainly switch to another video provider (Comcast!). It's not like this is a brand new issue, Netflix has held out for a long time. The Comcast / TWC merger is what finally did it, 30M combined customers is too big to risk on a public game of net neutrality chicken.
Surely they're still working on lobbying efforts to make it illegal for a company like Comcast to simultaneously have a monopoly on selling broadband and compete with services using said broadband. It's pretty obvious at this point that internet access is a utility and it should be treated as such.
""" But there's a silver lining for people who hate this deal: By the government's own standards, the Comcast—Time Warner deal should be blocked. To understand why, you have to know that there's a thing in antitrust law called the Herfindahl-Hirschman Index, or HHI. In very basic terms, the HHI is a way to measure the concentration of a given market. And the HHI is the test the government's lawyers apply if they want to see whether a proposed merger would create monopoly-like conditions. """
The Hacker News readership includes many people capable of starting ISPs. Comcast's large profits, which have been used to fund a string of high cost acquisitions, are a juicy target for potential competitors. Yet you never see discussion of ISP startups on Hacker News because the barriers to entry protect the current carriers. This wasn't the case in the 90s, and had Hacker News existed back then ISP startups would have appeared frequently.
The issue of telecoms not having competition has been the case for years. Back in the BBS days Jack Rickard used to harp on this in Boardwatch [1] magazine. I recall him saying something like, "Two guys with a screwdriver should be able to start a phone company."
Indeed. The late 90s / early 2000s were the golden age of the CLEC. There were certainly pain points (like getting copper pairs provisioned), but I remember not caring that my building didn't support Comcast Internet because I had an SDSL connection from Telocity and an ADSL connection from Speakeasy.
That said, most were either bought out, or went under within a couple years. It was a rough time for commercial SysOps in the mid-90's... not that I really feel bad for them. Times and environments change... The BBS and art scenes are still around today, but not nearly what they were in the early-mid 90's...
I miss it a lot... since most BBSes were based in a local area, there were a lot of get togethers, and you'd meet and talk to people you wouldn't necessarily do in today's internet, which is based in interest groups.
The cable industry has evolved into a bunch of regional monopolies that do not compete with one another.
The effect of breaking Comcast up AT&T-style would be... a bunch of regional monopolies that do not compete with one another. And then, as with AT&T, proceed to consolidate via M&A into larger regional monopolies... that do not compete with one another. Same as now.
Welcome to the American telecommunications industry.
That's totally backwards. Comcast's customers are paying for access to Cogent's customers, including Netflix.
It's not Comcasts responsibility if Cogent did not have peering arrangement or had paid for sufficient capacity to handle their end of the connection, that their customer was paying them to provide.
If it was, I'd love to provide Netflix connection much cheaper than Cogent. I'll provide it over my 10Mbps uplink, and blame consumers ISPs when everything falls over.
Since you've paid Comcast, they're now required to go to my brother's ISP and arrange more peering, right? Also, since you're paying, I shouldn't have to pay my brother, and he shouldn't have to pay anyone else, right?
Come to think of it, why isn't all hosting bandwidth free? Why do Azure and AWS charge for outbound bandwidth, but not inbound?
Netflix pays level3 for transit service (you pay your brother). Comcast pays level3 (your brother) for transit service. Notice how level3 is neither the source nor the destination of the traffic.
Netflix's traffic increases from, for example, 500Gbps to 1Tbps. Comcast's level3 ports fill up because netflix increased in popularity.
In general networks that pass a lot of traffic like this end up peering directly and avoiding transit costs.
Comcast has two choices. They can either pay level3 for another 500Gbps or they can transition the full 1Tbps to free peering directly with netflix. And yes, I consider it Comcast's responsibility to do one of those two things with their subscriber revenue, which, incidentally, most other ISPs in the US are doing.
Instead they are choosing option 3 which is, leverage their local access monopoly to force netflix to pay them, and protect their cable and xfinity services.
Let's do some simple math to prove your point.
Assume Comcast has about 10 Tbps of traffic at peak, single-counting all traffic that transits AS 7922 (Comcast's national backbone network). [1] Now assume that a full 50% of that traffic is originated by Netflix, and the market price for transit is $2/mbps. The percentage is likely quite lower than that, and given 5 Tbps of traffic volume and coordination between the two networks to reduce the number of miles the bits need to be hauled by both networks, the per-megabit price is almost certainly quite a bit less.
But even with these obviously flawed guesses, we're only talking about $120mm/year in costs. Netflix's worldwide revenue in 2013 was $4.3b [2]; Comcast's 2013 revenue from their US HSI business alone was $10.3b. [3] If 50% of Comcast's $10.3b business was in jeopardy, don't you think they'd find a way to absorb that $120mm?
[1] http://as7922.peeringdb.com/
[2] Netflix Q4 2013 earnings release, http://ir.netflix.com/results.cfm
[3] Comcast 10-K filed 2/12/2014
(edited: formatting)
This may be the cases for ISP-ISP connections, but every transit circuit I've ever purchased as an end-user has two elements, a Non Reccurring Charge (NRC) to set up the connection, and then a monthly recurring charge (MRC) consisting of, a port speed, and a 95th percentile metered usage charge.
e.g. I might get a gigabit port set up for an up front $5000 NRC, and then a 24 month contract for $2000/month port speed, and then pay $5/megabit @95th percentile of 5 minute averages sampled over a month.
The ISP samples the usage of my port every 5 minutes, (8640 samples in a 30 day month), drops the top 5% (432 samples), and then charges me the cost of the next sample (sample #8208) - If the sampled speed at #8208 was 1 Gigabit, and sample #8207 was 0 kilobits, then I am still charged $2000/month + 1000 * $5 = $7000/MRC for that month.
Cogent has always had a reputation in the industry for being the cut-rate player (to the point of other ISPs just dropping them as peering connections, resulting in companies literally not being able to communicate with their customers) - Netflix was trying to rely on Cogent to send data to their customers, presumably because they were charging significantly less than "Tier-1" transit-providers, and finally came to the realization that yes, you do get what you pay for.
None of this surprises me, and none of it appears relevant to the conversation regarding Net-Neutrality - this is a peering discussion, pure and simple.
Netflix does not send any data. Comcast customers request lots of data from Netflix. Significant difference.
Another example (just in the other direction): A backup provider needs to pay up big-cable because their customers are uploading a lot of data?
I'm willing to wager that backup providers haven't run into any upload issues with Comcast for this reason.
Each individual Comcast customer has in theory paid for a DS3 worth of bandwidth (modulo the "cap"), which should be a reality-check that some of your assumptions from that era need updating.
The defense (and it was always a thin one) of ratio clauses in peering agreements was hot potato routing. The ingress traffic is what you actually have to haul a long distance, so the agreement says "you must meet us everywhere we peer, and you must maintain ratio." Together, those rules mean each individual path will be asymmetric but the overall load on the WAN will be shared between the peers. It was always slightly silly, though.
Hot potato is now over. CDN's are desperate to do all the work and bring traffic as close to eyeballs as the monopolists will allow because the cost of actually building a network is irrelevant compared to the rent the monopolists are extracting. The monopolists are selling so-called "transit" to haul packets 1 mile. They're really charging for access to their captive eyeballs, not for moving bits. If you want to send a few megabits to Asia, fine, whatever. You can do it for the same price. They're not even mad, "glad we could make a deal," etc. The situation is nakedly broken and warped, and everybody actually in the business knows it.
Peering has always been a mix of market power, shaming, and political excuses ("ratio" was one). What we're now seeing is the shaming and excuses fall away thanks to apologist bloggers and weak politicians, and market power emerge as the only thing that matters. Compared to this power, traditional outdated notions of neutrality are irrelevant: there is no need to give traffic less "preferential" treatment when you can simply depeer them and then try to sell them so-called "transit" to go 1 mile. "Paid peering" == "well, how much money you got? Let's make a special deal just for you".
Without meaning to express either approval or cynicism, market power has always been the only thing that matters in this business. But CDNs have paid big ISPs special rates for years, because they really do put special strains on infrastructure. And my impression remains that that's what this particular battle is ultimately about.
When we do business with cloud providers or other third parties, we usually start with internet based access. When the relationship gets bigger or needs to serve a large portion of our base, we typically either peer with that provider (in the case of big cloud providers like Amazon, Microsoft, etc) or establish dedicated connectivity between our networks.
I think Comcast, Verizon, TWC, etc has a point here -- as a relationship grows with another party, you should have a more robust connection. Netflix hasn't wanted to do that -- it prefers to use a shitty ISP (Cogent) for cheap, and augment it by co-locating content on end-user ISP networks for free. Services like Netflix and Youtube stretch the net-neutrality argument, because they aren't good citizens.
That said, the end-user ISPs cannot be allowed to discriminate, which their monopoly power will almost certainly enable them to do.
I think it's ridiculous and self-serving for internet companies to complain about this. Getting bits to customers is a critical part of any online business. You can either do it yourself, or pay someone to do it. Does Amazon complain that UPS charges them to deliver packages to customers?
Really, the point of the Internet is that you can communicate with other Internet users without having to worry about what ISP or network they are connected to or what arrangements govern that connection.
The more apt analogy would be if UPS decided to make Amazon packages ship super slow unless Amazon payed extra fees to "upgrade" the shipping speed back to average.
You could make the case that Comcast should have upgraded their peering with Cogent. That's the case in almost all of these disputes. But Netflix blinked first.
If you want to make useless shipping analogies, it's like Amazon ships using ABC to a rough wilderness. ABC uses simple cars that don't handle the terrain so they contract the local government to deliver packages in that area. You could say it's the local government's responsibility to fix the roads, or ABC's responsibility or anyone but Amazon's. Good luck.
This is simply the case of a provider, discovering that Cogent provides poor transit services, realizes that you actually have to pay in order to get good Internet peering. This has always been the case on the Internet.
As long as Comcast treats all traffic the same regardless of source, destination, packet type, etc.. I'm fine with their behavior here.
I guess the question is, did all other traffic see the same drop? And what changes in their network would cause this?
Netflix customers, in this situation, can point their fingers at Netflix (who, in turn, can point them at Cogent).
If Cogent provided 1 Gigabit of transit to Netflix, but didn't have circuits with which to connect Netflix to their customers, well, It will be interesting to see what kind of Liquidated Damages Netflix wrote into their contract with Cogent. Maybe it will be enough to pay their peering fees with Comcast.
I agree in spirit, but how do you make the distinction? If Cogent has poor connectivity to the wider Internet, is it on Comcast to fix that?
In your opinion why doesn't the same responsibility apply to Cogent?
Tata is also spectacularly bad with Comcast.
I guess you've never heard of VPNs, then?
Edit: Having read more of the non-paywalled version, I have to say I think they're shooting themselves in the foot, even if Comcast was offering a better price than Cogent. It sets a terrible precedent, and gives Comcast the leverage to bump the price extortionately at the end of the deal.
Kipling has a cogent explanation of it in poetry form:
http://www.poetryloverspage.com/poets/kipling/dane_geld.html
I suggest we treat them accordingly.
;]
The term you should be looking for is “paying the danegeld”:
That if once you have paid him the Danegeld, you never get rid of the Dane.
— Dane Geld, Rudyard Kipling
This is the creation of an economic moat (a term coined by Warren Buffet.)
The gatekeeper creates a toll and the tradesman agrees to pay the toll. This is an entry cost to new market players and therefore reduces competition. This toll may be increased as the biggest player (Netflix) grows in size and this benefits them, as eventually new startups have $1+ million startup costs and nobody is willing to invest in that cost-payoff scenario.
This is a win-win negotiation for both Comcast and Netflix but a loss for consumers. Unfortunately it's just how the game is played.
Whoever has the market right now has it extended for a long time.
Generally the bigger you are, the more you will want to avoid volatility as big tends towards fragile - a big company will prefer static contractual obligations to another big company over being attacked from all sides.
(I am unsure however how this game plays out in the later stages if they lose leverage and entry costs increase to the point that it begins to suffocate the market.)
But I framed it too specifically. My perspective holds true at the level of large organisations wanting to take punts towards similarly large markets. Signing the money off is going to become more and more difficult when the access cost is so high, and barely anybody will want to take the reputational cost of failure to enter the market.
Everything we know about Netflix is that they're a data driven company. Chances are it's far more likely they have seen viewing rates drop for Comcast customers who were affected by the slow-down.
If a typical end-user sees their internet working normally, bu see slow-downs selectively to streaming to sites like Netflix, chances are they're going to wrongly attribute the slow-down to the service and not their internet connection.
Still, perhaps Comcast will deem all of the companies big enough to compete with Netflix large enough to pay the tax.
We are watching the ascension of the new barons. They will hold their privilege with an iron fist, for as long as they hold it, they define the playing field. They would be foolhardy not to.
The barrier to entry will indeed rise, as peering costs will end up passed on to content producers rather than consumers - consumers will not accept the cost rises, and market competition on the ISP side will ensure this doesn't arise - but on the supply side, large players can and will pay for an advantage, and small players will either be subsumed or destroyed. New entrants will need to be well armed indeed.
Now, this isn't quite the same as the feudal picture as previously described, as the barriers will at first be monetary rather than cultural, and have not yet had time to thoroughly entrench. If this is allowed to persist as is, however, without resistance or concern, we will indeed find ourselves in a entrenched technocracy of the foulest dimensions, with very little recourse for anyone without access to vast capital.
1. Netflix is currently using Cogent
2. Comcast offered them a direct connection to their network via paid-peering.
3. Netflix agreed and will move over.
4. Nothing about net neutrality was mentioned in the article.
This deal is being referred to as paid-peering. Peering is a practice performed at ISPs where two ISPs hook up infrastructure and agree to maintain their sides without charging for bandwidth. Peering is a common practice among the industry, and content-providers are always looking for ways to peer with more ISPs because it drives down their overall cost of operations.
The article says nothing about net neutrality. We don't have enough information to say whether net neutrality was harmed in this deal. We can only speculate at this time. Hopefully we will hear more from Netflix on this.
[edit to fix list formatting]
Is is normal. ISPs are not equal. There's a reason you pay Level(3) a lot more than HE.
Comcast won't charge new streaming providers from the get go. That will stunt its own business opportunities. Instead it will prefer to let them grow to the point where they can claim network congestion issues. At that point it has a "justifiable claim" as well as more negotiating leverage.
If you're a small provider, you tend to buy transit from one or a few providers, and pay per bit in whichever direction has the most traffic.
If you're bigger, maybe you can setup peering. If you have relatively balanced flows, and meet other qualifications, you can usually get free (settlement free) peering. If your flows aren't balanced, then the tradition is that whoever is sending more pays. If the networks you're establishing paid peering with are cheaper than your transit provider (usually the case), then peering is still a win for you, and a win for the other network.
If Netflix wants to get settlement free peering with Comcast, they need to balance their traffic to Comcast (maybe partner with a backup service, or web crawler)
This has nothing to do with net neutrality. Comcast is not throttling or discriminating traffic in any way (at least not publicly). This is a peering dispute settlement.
Does this affect competitors to Netflix? Depends on how much volume they have. If their volumes are low, they can still reliably use transit providers more economically. As their revenue increases, they'll need their own backbone network, but their ability to pay for it goes up too.
Is Comcast big, bad and evil? Probably. But not in this case. Comcast dealing directly with Netflix is good for both parties. They are entitled to charge Netflix to peer directly because most traffic is going to flow into Comcast's network from Netflix. They also probably had a case negotiating away from settlement-free agreements with Cogent for the same reason. Again, none of this is out of the ordinary, ASes negotiate peering deals with each other all the time based on the volume and direction of the traffic they exchange with each other, and this is all fairly unregulated.
Net neutrality is irrelevant in all of this, it only matters if Comcast actively discriminates between traffic at their end. This deal is not connected to the ruling.
No, what happened was Comcast discovered another place they could fuck over the customer - by charging for peering even though it benefits their customers to not do so, and decided to spin it into some way that makes people feel like Comcast was somehow being injured by Netflix under the old settlement-free peering model.
They will now be DOUBLE BILLING for the same packet, a cost that WILL BE passed along to the netflix customer.
Savings(if any) will not be passed along on the comcast side
Personally I do not believe there will be any real saving for comcast in the first place, so there is nothing to pass along, it is 100% profit for them
This made me laugh seeing as every one of Comcast's customer ISP packages have 5 times higher download speeds than upload speeds.
It's almost as if Comcast is forcibly causing an imbalance of traffic so they can extort more money out of Tier 1 providers.
As part of the Open Connect Project, Netflix has deployed servers which are hundreds of terabytes in size in strategic locations all over the work and they contain ~60-90% of the data required for that particular country.
These devices either live in the ISP (which is what Netflix wanted to happen with Comcast) or live in an internet exchange where multiple ISPs can connect to the Open Connect devices.
There are 3 core benefits in doing this:
- Better Streaming due to fewer hops to the client
- Netflix avoids rate limiting (if they're inside the ISP Network)
- Peering is free (usually) and is essentially where two ISPs connect their infastructure and agree to maintain their own sites without bandwidth charges. It's common throughout the industry and content-providers are always looking to peer with ISPs to drive down their overall costs of operations.
The only other explanation is that they may actually like this, because they know it would slow down a lot of their smaller competitors coming after them. It's even stranger when you think that customers must've blamed Comcast for their poor Netflix experience lately, not Netflix.
You can't really guarantee uncongested paths at all times to the "whole Internet".
And, let's face it - anybody who has ever used Cogent for transit (usually at 1/3 to 1/10th the cost of other providers) knew what they were getting into.
There ya go.
It's like a hostage situation; you say "yeah I don't negotiate with terrorists" but when you see your wife/kid/so threatened and the money is just there in the bank it's hard to hold on to that posture...
Would you please enlighten me as to why an ISP should be required to provide access to a resource they aren't being compensated for?
Cogent should pay Comcast market peering costs for the unreciprocated bandwidth they send to Comcast. That's what this whole dispute is about. Arguing that Comcast's refusal to build out and maintain significant infrastructure to support a peering imbalance imposed on them by a peering partner via abuse of a good-faith concession, at their own expense, is somehow "taking hostages" is just silly. Cogent oversold.
I think that Comcast sucks as much as the next guy, but I really don't think they're the bad guy here (as much as it pains me to say that). It looks that way to consumers because Comcast is the only contact affected users have with the whole industry (how many people who have been upset at crappy Netflix quality know who Cogent is, or what a transit agreement is?), but that doesn't mean that's actually an accurate perspective.
No, I don't think Comcast is being abused at all and its much more likely they just don't give a shit what their customers want because they know there is nothing the customer can do about it and so they let service degrade in order to extort more money.
If this were not consumer-bound data, then Cogent would likely already be paying for the disparity. As it is, it's a good guess that Cogent has refused to compensate Comcast for the additional load on their network build-out and maintenance costs, instead relying on the good-faith consumer data exception to exempt them from paying for it. Comcast's response to this was to steal Netflix's business from Cogent, thereby alleviating the uncompensated load that Cogent was placing on their network and improving Netflix's deliverability.
There's no such thing as a free lunch. If you're a service provider, you're paying someone for delivery of your bytes. Netflix just cut out the middleman here. It's not like Netflix didn't already have to pay for the bytes they delivered - they just changed who they're paying.
The only solution, is to cut off this excess profit that has become the lifeblood of this systematic abuse.
And where these oligopolies have essentially captured the market to the point of excluding any and all competition: End them, legally. Break them up. Specifically legalize and perhaps even foster municipal broadband.
There may be no ultimate, static solution. But there are definitely next steps, and they are overdue.
I am already paying Comcast for access to the internet! Why does Netflix have to pay on top it?
That said, I have Netflix and Comcast and I have never experienced any issues.
Edit: Actually I realize, this is an agreement to allow Netflix to colocate some of their servers at Comcast hubs. So it is not actually that bad.
"Neutral" is a vague term that in this circumstance conveys no actual meaning.
Settlement free peering is the exception, not the rule. Almost everyone pays for access to the Internet, whether it's buying transit or peering with settlement.
Nothing new to see here, move along.
The reality is that it was likely cheaper for Netflix to just pay for peering with Comcast directly.
What are you referring to? Comcast and Netflix just started peering a few days ago.
One tangent that I don't know where else to ask is: why would an ISP agree to peer with Netflix, if Netflix weren't paying them. Reading about openconnect makes it seem that Netflix wants free peering.
My understanding of the situation is as follows. 1) Settlement free peering agreement are equivalent to free peering agreements. 2) Companies agree to these if they send roughly the same amount of traffic in both directions. So I'll agree to peer with you if I want to send ~N GB of traffic through your network and you want to send ~N GB through mine. 3) (and this is where I'm really unsure what I'm talking about) Anyone who peers with Netflix deals with handling traffic from Netflix to it's customers. But what traffic goes the other way? What traffic would an ISP send through Netflix network to make the relationship mutually beneficial?
Isn't it just the normal way of things that content producers pay ISP to send traffic through their networks?
Thanks for any clarifications!
[1]http://www.telecompetitor.com/level-3comcast-dispute-revives... and
[2]http://www.telecompetitor.com/behind-the-level-3-comcast-pee...
Your understanding is absolutely correct - this is not a Net Neutrality issue, but a Peering Dispute, plain and simple.
Also, another tangent: I'm wondering what happens when Netflix installs one of their cache servers at an ISP data center. If 10 customers want to stream House of Cards and that's 10GB of data, I'm assuming Netflix has to end up paying somewhere for 10GB of upload bandwidth.
But if the Netflix cache server was in a nearby ISP data center, would Netflix only pay for 1GB of bandwidth (to upload to their servers), and not pay for 10GB of bandwidth as the customers hit the cache server? I'm assuming that the ISP makes some of it back by charging Netflix to place their servers locally, but it's still cheaper then paying to stream all the bandwidth.
If an ISP were completely focused on maximizing it's profit and not on what Netflix quality was to it's customer, does the ISP have any incentive to install Netflix cache server?
Thanks again for clearing up these questions!
The ISP has a big incentive to have Netflix increase their quality wherever there is strong competition for Broadband Internet. The more competition, the more incentive.
Here is an excellent article on transit: http://blog.streamingmedia.com/2014/02/transit-works-costs-i...
So much for net neutrality. Now ISPs have another way to make more $$. Keep those profits higher - push smaller streaming services out.
Motherfuckers.
[1] http://gigaom2.files.wordpress.com/2013/05/usbroadbandsubscr...
This may mark the end of net neutrality.
I think links with Google as a referrer show you the whole article.
1) Collect evidence of network tampering/filtering
2) Get agreement in place to pay for better service
3) Collect the rest of the evidence that network filtering was in place when it gets lifted.
4) Simultaneously hit comcast up for breaching the neutrality agreements, while notifying all comcast customers that comcast is forcing netflix to charge its customers more for access.. that it might have to be along to them to pay... but we are working on ways yo prevent it.
Try and squeeze comcast from both ends at once and public embarrass them, while trying to keep the higher ground, looking like the peoples hero, under dog etc...
I doubt many companies would take that risk, but that what I'd try.
I'm curious to see how this plays out because the endgame for the internet is free-fast-failsafe. I wonder how long it will take for companies to litigate themselves into irrelevance. I actually feel kind of bad for netflix in this case, because when you make a deal with the devil, he always wins, and I hate to see them go someday :-(
Unfortunately, this (and most other similar) sites use Flash, something I'm trying to avoid[0]. They also tend to have only low-quality, grainy video (not that Netflix is much better even when buffering isn't an issue).
There is also the issue of these sites/videos coming and going. If they host the files on 3rd party platforms (like this site is doing), then once the site is popular those videos quickly get taken down. Then you're stuck clicking through trying to find that one site (if there's even one left) that still hasn't taken down the video. OTOH, if the files are self-hosted, then when the site becomes popular, it can no longer support the bandwidth costs, and goes under.
Torrent sites, however, have lasted for years. I've been using the same private torrent site for nearly a decade. It's reliable and fast, and it has a wide selection of content, both old and new. And I never have to deal with Flash.
Prior to the agreement, the traffic was routed through Cogent, and Comcast didn't want to increase the interconnection bandwidth with Cogent.
I don't see this as a blow to Net Neutrality, assuming the deal is as described.
Monday evening there will be a vote on net neutrality. CONTACT YOUR REPRESENTATIVE
http://www.savetheinternet.eu/
(dutch) https://www.bof.nl/2014/02/21/maandag-d-day-voor-netneutrali...
Explain your concerns to them.
We paid for the network, anybody selling transit across it should be able to use it.
Say something like "we can't provide a great experience so we think you should have no experience at all because it wouldn't be fair with us and with you. we are sorry." and make it explicit that it's because of the ISP they are using.
The ISP think they are selling gold when they are more likely selling a commodity. Sure, at some locations it's indeed like gold, but how much of that is artificially enforced?
These pipes aren't worth much if there is no content flowing through them.
The whole net neutrality discussion is a power play between content providers and fiber owners, nothing else. The discussion about freedom and competition is awesome (and right!), but at the end it's these two huge groups fighting each other while we are screaming "think of the children!".
What Netflix would be losing if a subset of their customers on provider X is already getting crappy service and it can't do anything about that? At some point they will have to decide what to do about all those complaints and refund requests.
There's no net neutrality issue in this discussion at all. Comcast is simply declining to upgrade its peering with Cogent. How is that net neutrality? If Netflix comes to me tomorrow and decides to buy all their transit through me, is Comcast somehow required to give me lots of free peering?
Netflix might be the majority(?) of Cogent's traffic, and Comcast might decide that it's not worth upgrading the connections for them. But as long as Comcast isn't discriminating traffic on their end (via some traffic shaping mechanism, I'd guess), then it's simply a matter of "Comcast thinks Cogent has enough connectivity, and sees no reason for them to upgrade the connections while getting nothing in return".
Their customers may disagree, but it up to them to not buy the service. (Perhaps the real problem here is lack of competition in the ISP market?)
I was commenting on net neutrality issues and that was obviously not related here.
This has nothing to do with you having to pay more to stream other services. It has to do with netflix finally deciding to stop placing blame and further their own product.
I'm all for net neutrality, but I think Netflix should've been paying some usage fee into a communal fund that is off-limit to ISPs, but can be used to improve broadband quality for everyone.
It is not as simple as you think
Oh wait, I buy from a friend who runs an ISP. Turns out, he only has a single 10G connection for all his customers. Can he call Comcast for a free peering arrangement now?
That's not how things work.
I recall the federal government providing billions to the cable companies in the 90s for high speed broadband that never materialized.
I was surprised that when I cancelled I didn't even get a single form asking why.
Netflix is $8 a month for all you can stream, so I suppose I can't expect any more than a poor library of B movies and old TV shows.
It's not up to me how Netflix and Comcast sort out their business disputes. Netflix was not delivering service that was worth their measly $8 a month.
hollywood pressures comcast.
comcast shakedown netflix.
netflix offers money to comcast, comcast and hollywood is pleased.
so far it sucks... but then:
netflix pass that as tax to user.
users flee from comcast.
new providers appears.
hollywood cant shake them down.
for the first time, ISP in america have a truly competitive market.
the end.
I'm thinking of switching to Sonic.net - they seem to respect Net neutrality. Does anyone have Sonic.net experience pro or con ??