Hacker News Reader

Comment by jmuguy | Hacker News Reader

My understanding is that its not an issue of the cost to "upgrade" the interconnection infrastructure. That cost is likely negligible. What's happening is that VZ is refusing to do it, and allowing service to degrade, unless Google pays X. So yeah, basically extortion. Google has already paid for bandwidth, and this is traffic that VZ's paying customers are requesting.

View on HN

let's do a little math..

the router ports on both sides, let's say you are using 4 ports across 4 fully loaded nexus 9508 (let's call those $500k per incl. optics, so $2M / 384 x 4 ports / 36 months = $600/port/month x 4 = $2400/4-ports/month)

Depending on how the connection between the two worked or was paid for, it could be $1000-$20000 for 2 pairs of fiber per month depending on distance. Let's take a middle of the road $10k average cost.

So for 10Gbps peak redundant capacity, you are at $12,400 per month. Netflix 4k stream is about 16 megabits, so you can fit 625 4k streams in $12,400 of cross-connect capacity. If you don't share costs, that's about $20 per user to support Netflix's business model that people seem to think it's Comcast's responsibility to pay. That's not even considering Comcast's last mile distribution cost or paying any salaries.

Streaming video services soak up a lot of network resources and putting all of the cost on ISPs is going to increase your ISP's cost and price. How many low-income families do you think would lose broadband if it went from $50/mo to $100/mo?

That's assuming that we're talking a connection streaming a video from Netflix's CDN to the customer's device. When in reality, Netflix offered to put a CDN cache box in Comcast's DC which would largely reduce the problem, Comcast turned them down.

That, alone, tells me that this is not a congestion or a cost problem, it is a bull-headedness for the purpose of rent seeking problem.

And as to the "last mile" costs, those are what Comcast subscribers are paying Comcast for. They act like this traffic is unsolicited noise, when in reality it's why they're being paid anything by anyone. I say don't charge people for x megabits down if you don't intend on them using it.

My understanding is that Comcast offered Netflix commercially reasonable terms for colo and cross-connect and Netflix opted for public interconnect via an exchange, which alleviated the bandwidth constraints caused by Cogent taking Netflix's money and then refusing to comply with their cost-sharing peering agreement with Comcast. Calling the Netflix CDN boxes "free" is totally misleading because Comcast would still need to pay for the router ports to carry the traffic internally, space & power, and management assistance. That deal works for a small ISP that pays for most of its bandwidth but not a large ISP that uses significant amounts of shared-cost peering. If Comcast is required to host those boxes "for free" what of every other CDN that they previously had agreements with?

I agree that Comcast marketing "up to X megabits" is misleading and those chickens have come home to roost with people thinking that their last mile means they should get that speed to every point on the Internet 24 hours a day. I have been negotiating commercial bandwidth agreements for years and even from a tier 1 you can't get that guarantee in a contract.

Calling the Netflix CDN boxes "free" is totally misleading because Comcast would still need to pay for the router ports to carry the traffic internally, space & power, and management assistance.

Which is still less than the cost of doing a proper interconnect - they chose instead to let it degrade and play semantic games instead. It also puts the lie to their complaint that it was ever about congestion. The correct response to people requesting a lot of traffic from X is to ensure that traffic is delivered efficiently. That is why their customers pay them.

Back in reality the huge ISP's provide network services at a huge premium and make a very substantial profit. If we cared about providing affordable service to low income families we would let municipalities sell access to existing fiber networks not trying to prop up an industry providing bad service at monopoly prices.

Your numbers are like historical fiction based on reality but not an accurate portrayal. Do you work for verizon or comcast?

Right, and Netflix and Youtube are charities, give me a break with that argument.

since you INSISTED, I have no personal interest in any ISP

If you're going to refute my numbers please provide your own.

Source your numbers. Most specifically that netflix which can provide the entirety of its service for 10 per subscriber actually costs providers $20 per subscriber.

That's not $20/user to support Netflix's business model, that's $20/user to support Comcast's customers' usage model.

If you think this is unfair, then the answer is to get rid of bullshit "unlimited [but not really]" connections and charge people based on what they use. Stop making low-impact users subsidize people who stream HD movies 24/7.

My ISP keeps on trying to get me to upgrade to "up to" 100 Mbps service. What in the world do I need that kind of bandwidth for except to stream videos? Shouldn't they disclose that if I upgrade I can't actually use the bandwidth unless the video provider pays them too?

That does seem to be a trivial point. You're not going to get 100Mbps unless the entire path between the endpoints can push that speed.

You're getting a 100Mbps connection to the edge of their network and they have peering agreements, what else should they be doing?

Eh, that's getting less and less of an issue. My nominal 105/20 internet connection from Comcast Business rarely pushes less than 80mbps download speeds, and from any major provider, CDN, regularly hits and sustains 120mbps.

> what else should they be doing?

They should be peering freely with anyone that is responding to requests from the ISP's end users.

>that's about $20 per user to support Netflix's business model that people seem to think it's Comcast's responsibility to pay.

So what is my $80/mo Comcast Internet Service supposed to be buying me?

Lobbying.

Seriously, though, could someone do a traceroute and tells me who pays for each hop?

Basically you (the customer) pay for the last mile and Netflix is getting extorted into paying for the rest.

Not just the last mile but the peering agreements that your last-mile ISP has with either a backbone provider and/or other networks.

Telecom and Internet lobbying aren't all that different: https://www.opensecrets.org/lobby/indusclient.php?id=B09&yea... https://www.opensecrets.org/lobby/indusclient.php?id=B13&yea...

Alphabet spends more on lobbying than Comcast.

Those numbers aren't even close to accurate. A fully loaded nexus 9508 (assuming the 10Gbe ports you listed) is ~300k LIST. I'd be shocked if a company that orders as much as Verizon does would pay even half of that.

As for the "distance" - they are sitting next to each other in a data-center. The distance is likely measured in tens of feet and the cost is likely a fixed cost of a couple hundred dollars. The optics on either end if they aren't close enough for twinax would add a couple more grand.

It's also ridiculous to use 36 months as your payoff date, they are running equipment a heck of a lot longer than that. The line cards and supervisor modules might get swapped out, but I'm willing to bet Verizon keeps their chassis level switches for 7-10 years on average.

So now you're at about 4$ per user to support Netflix traffic.

Now throw in some standard oversubscription math, and you're at 40¢ per user. 625 simultaneous streams can probably serve roughly 6,250 customers, given that half the customers don't use Netflix, and the half that do, use it at different times of the day.

Also the proportion of customers who needs 4k as opposed to 1080p is negligible. Most consumers don't even have equipment capable of displaying 4k content.

Netflix in 1080p uses 4.22Mbps or sometimes 7.15Mbps, so we're now at 15¢ per user.

Wouldn't they cross connect at 40 or 100GBe?

They would most likely given the cost per gb would be significantly reduced, but I was just going with his stated scenario.

Hold on a second. Your reasoning has some issues:

1) The peering happens in Internet exchanges, there's no way a peering connection in an Internet Exchange costs 1000$, your range is simply ridiculous. Usually, it is fixed cost 1K or less depending on the locations of the router

2) Yes, you have to pay for bandwidth between the Exchange and your users. But guess what? That's what your users are paying for.

3) Not everyone stream movie in 4K at the same time. So you could optimize your bandwidth usage out of the exchanges. It is called "multiplexing".

Yes, the rest of the chain could be expensive, but that's what your clients are paying for. Net neutrality is about transparency. I have no problem in understanding that my 40$ connection will perform differently than an 80$ one. But you should compete in giving me the best service (where best is best for me, not universally) at the lowest price point in a way I could easily compare prices.

The problem, with my statement, is that implies Broadband Internet is a commoditized service. And that is all the battle about Net Neutrality, ISP doesn't want to act like commodities, they don't want to be your P&G or your ConEd. They don't have the structure to compete and instead of changing for the new market structure, they are fighting back. What they are doing is rent seeking. Good for them, bad for the economy...

I acknowledge that the costs are ballpark and not exact, but I was responding to a post that called the costs "negligible" - I was simply trying to quantify a slice of the cost and demonstrate that it can be a significant fraction of your bill. Of course your bill ALSO pays for the last mile infrastructure and connectivity to the hundreds of thousands of other routes carried on the internet. But it's clarifying to see that the data interchange component, which people are calling "negligible" actually is a significant portion of the cost.

My numbers have been called ridiculous by a couple of people now but they are based in personal experience, albeit a few years old, and nobody has posted any other cost breakdowns that demonstrate an understanding of the industry and the costs involved, just "those numbers seem really high!"

The difference between bandwidth and power is that data is NOT a commodity like power is, power is fungible and can be drawn and combined from a number of sources to fulfill the demand, but the dilemma with ISP bandwidth is that in order to satisfy customers the ISP must ensure adequate bandwidth to each individual content provider, and this is a much harder problem.

The problem is your numbers are terribly wrong in multiple ways. I don't think people have mentioned that not every subscriber is online simultaneously or that most are still watching 1080p.

Your argument is at least as wrong as mine, though, while you seek to reduce the numbers, you are not also seeking to increase the scope of the cost from JUST the hardware depreciation and monthly fiber cost of the interconnect to the actual cost of the interconnect. The point is, it's not even close to "negligible"

Not to mention if you're saturating a 10gbps connection, Netflix will throw as many OpenConnect appliances at you as you'd like. Yes, they're not "free" when you factor in racking, etc, but local bandwidth is cheaper than peering, even factoring in your valid comments here.

It's not Comcast's responsibility to support someone else's business model.

It's also not Netflix's responsibility to support Comcast's business model if they can't afford to provide bandwidth that customers desire and are allowed.

It's Comcast's responsibility to support their advertised usage sufficiently. If they say that they support 100mbps then I should be able to get 100mbps from wherever I like. They're going to have to pay for that uplink somewhere, whether it's Netflix or Google or Zayo or Level3 or whoever.

I agree, this is the crux of the argument. Broadband ISPs have been guilty of this for some time. Mobile players are generally getting a little more leeway by capping total transfer per month and not talking about rates as much. And T-mobile's "Binge On" product has been mostly hailed as pro-consumer, even though it does shape video down to SD bitrates and is a direct violation of the "paid prioritization" tenet of net neutrality as described in FCC's directive.

> If they say that they support 100mbps then I should be able to get 100mbps from wherever I like.

Precisely. Not "from wherever we've got a 'preferred' agreement or kickback only".

> that people seem to think it's Comcast's responsibility to pay

Funny, the upwards of $50 to $200 I pay Comcast per month should go somewhat to that.

Besides, how many users are streaming 4K content, on what little 4K content Netflix has (I'm going to wager that we're looking at about 5% or less).

At 5mbps for 1080p, peak, now we're at 2,000 users, and $6/month.

Is it really that onerous for most users of Comcast to expect that $6 of their say $60/mo cable bill goes to Netflix?

I'm also not sure why you're populating 4 ports and then only talking about 1 10Gbps connection, when in reality (though I'm no expert), that's probably 4 10Gbps connections. Accounting for the cost of 4 populated running ports and then only talking about the capacity of 1 of those ports when calculating cost/megabit seems misleading.

That's what splitting the bill means.

You don't pay "for the bandwidth", you pay for a link between two ISPs (Google and Verizon). The bill between then should be split, otherwise the traffic will have to pass through somewhere else, and that will cause congestion problems.

Google has most certainly paid for bandwidth/traffic from their end. Whether that's to a CDN or some other arrangement, and there's likely already peering agreements in place. In these cases what Verizon is doing is asking for money above and beyond what it actually costs to handle the interconnection. They use declining service to their own customers as leverage. Its ridiculous, and even more worrying when you see ISPs and content providers merging as is the case with VZ buying Yahoo and AOL, Comcast and NBC merger, etc.

Reply on news.ycombinator.com