Netflix, Long an AWS Customer, Tests Waters on Google Cloud

For years Dell would let it "leak" out to the press that they were looking at supporting AMD chips. It would happen every year like clockwork. Then after a month you'd hear that Dell and Intel had come up with a new deal to remain Intel exclusive.

Really playing these companies against each other would mean dynamically balancing your services across the providers based on pricing

If it's a final decision, then most of the cost is in the future, far beyond the reach of the initial quotes.

I've seen too many companies turn extractive (including, of course, that same Microsoft) to ever use the words "final" next to "platform decision".

They'll play a few games with contracts to spread the costs around (e.g. giving n% discounts for m years of reserved capacity). You probably won't get much just by saying "I can just go to GCP" and asking for the world. The more you spend, the more distinct services you buy, and the longer you want to commit the steeper the discount. I've seen up to 10% but I'm sure they do more for bigger customers.

Seriously? or are you a member of AWFUL - Americans Who Figuratively Use Literally ;)

No http2 support is increasingly limiting

loaf -> load, lol

Antiquated doesn't mean unstable or unusable. It means old fashioned. It's not the new hotness.

To answer your question directly, AWS's poor spike handling has really made me distrustful of them. It's not that AWS's support staff is bad (actually, they're phenomenal given the difficulty of that job) but... the fact that I can be taken down by a big scale event because I can't pre-scale my ELBs without support help that takes hours is kinda annoying.

Now, I've been TOLD by ___________________________________ this is actually because it's a pain in the butt and they're working to fix it, but AWS's budget seems to favor new shinies over iterating on the existing environment. Look how long it took us to get a new and less buggy AWS dashboard and that still hasn't propagated to all the internal services.

I could go on. For example, my experience is that GCE's and even Azure's I/O and networking is way faster, more consistent and more reliable than AWSs. I host all my dev boxes on GCE because using an attached drive (so I can de/re-provision to save on costs) is much less agonizing. Also: google has some AMAZING database products that category kill whole infrastructure tasks. It's a very good kind of vendor lock-in when they can say, "You can't move away because no one else has found a way to provide our feature set."

ELB is the worst LB out there.

Why?

He led the largest migration from on premises to AWS. Who else would have more experience?

No way, if I was CTO of a big company (or, say, whoever at Pentagon is in charge of IT), I would love to speak to the guy that shifted Netflix away from in-house datacenters to "The Cloud".

And now dropbox owns those datacentres. ;)

You're ignoring the fact that he spent a few years at Battery Ventures in between Netflix and AWS. Adrian is a well respected professional in his field, has been around for decades, and nobody holds any negative perceptions of him.

He wasn't hired by AWS because he migrated datacenters to the cloud. He was hired by AWS because he created this: https://netflix.github.io/ - Netflix's OSS, which has allowed them to attract so much great talent, was a direct result of Adrian's passionate open source advocacy. He's doing the same thing at AWS now, and has already accomplished great things like open sourcing all of the AWS documentation.

Disclaimer: work at AWS.

+1 regarding RDS. A company I spoke to is moving from GCP to AWS, with their biggest pain point being Cloud SQL is WAY less flexible than RDS. While GCP's k8s offering is better/easier to use, they still have some catching up to do.

To drive the second point home, GCP recently released VPC Flow Logs, while AWS has had it for a good while now.

They have data centers of their own and distribute all of their own content. AWS powers the UI requests and their back office number crunching stuff. Everything after you push play is done via Netflix's own hardware.

They _had_ their own datacenters, and decided to move their control plane to the cloud (AWS) to avoid having to run the those datacenters themselves.

https://media.netflix.com/en/company-blog/completing-the-net...

This was a big deal in 2008 :)

(Maybe there's an even bigger size where it no longer makes sense for them to outsource this, but bidding GCP and AWS against each other seems like a reasonable strategy for their use-case).

While that might be the case, I wouldn’t be surprised if they let teams play with k8s as well. Besides, one should always look at these choices objectively; if Google + k8s is a better deal overall than AWS & Mesos, then you shouldnt let your earlier investments cloud your judgement.

I know both Kubernetes and Mesos quite well. To be frank, they're on two different levels - and I assume the smart(er than me) guys at Netflix know that as well.

Microsoft pulled out of the music space, they're still selling & renting movies/shows and content services are a core feature of azure. Netflix uses its own POPS for of its content delivery.

Isn't it better to be in bed with your competitor so as to have a solid claim against being declared a monopoly?

Original YouTube content is really not a huge part of YouTube's offerings. Even if it were a larger component, Google wouldn't jeopardize a large hosting account just to score points against a competitor.

Amazon, on the other hand, will partner and make nice with competitors until they're ready to abandon them. Amazon is planning on supplanting FedEx and UPS and already has its own fleet of cargo planes. Amazon is great for business until they decide they don't need you anymore.

No one would dare try to sell a set top box in the US that didn't support Netflix.

Google also may have a competing service if they ever figure out what they want YouTube Red to be.

I use Amazon Video constantly now that Netflix is mostly/hugely populated with Netflix Originals only.

Let's be honest, even in the public cloud there are very few orgs that can survive regional failures and most people solve locality through CDN's for content delivery which means a private datacenter isn't really that different from public cloud for most orgs. Also, when when orgs move to the cloud they don't fire their networking team, they still have deskopts/laptops/servers/security devices/printers and large lans/wans to connect / peer already and these teams could staff up for private datacenter without really adding additional costs that aren't already incurred as part of doing business anyway. Our private datacenter is a few thousand servers leased space in a colo and we have one "datacenter dude" who handles hardware issues and our network team spends 90% of their time still on corporate infra vs actual supports with colo space. One can take some up front engineering/planning costs to scope out a private datacenter for about the same cost as it is to scope/plan VPC's and peering/connectivity/vpn services.

Everything you'd have to pay for, Amazon or Google has to pay for for you... plus profit margin. At Netflix scale, it's almost impossible for someone else's cloud to be cheaper than running your own, and hiring the necessary staff and acquiring the necessary space and hardware to do so.

I honestly have to wonder if the level of Amazon's discount for Netflix is absurd to the tune of "we lose money doing this" just to retain them. AWS may need Netflix, to make serving it's smaller clients economically viable, more than Netflix needs AWS.

That's a good distinction. CoLo providers are pretty reasonably priced and most of the larger ones are good at what they do.

There's a HUGE scale difference between a CoLo and your own -- as soon as you start having to negotiate diesel supply contracts yourself things get crazy fast.

I think this is a crucial distinction. Netflix has a relatively unique architecture for their content delivery system compared to other online video providers, which is worth considering in context of this article.

With Netflix's Open Connect (https://openconnect.netflix.com/en/) system, most residential Internet Service Providers have appliances at the regional point of interconnection. These appliances are inside of or directly connected to the ISP's network in the region. Back in 2016 they had appliances deployed to over 1,000 locations. Content is pre-positioned during off-peak hours to the appliances.

Previously Netflix has been open about how they use AWS for "generic, scalable computing" including "all of the logic of the application interface, the content discovery and selection experience, recommendation algorithms, transcoding, etc."

At the time, they claimed that one of the advantages of doing this was the ease of use and growing commoditization of the “cloud” market, so it makes sense that they are evaluating alternative cloud providers for these commodity services.

I've used most of Hashicorp's tooling - Consul, Nomad, Vault, etc. - and I consider myself to be a Hashicorp fanboy. But if you're spinning up a bunch of VMs "in the cloud" and that's all you're doing, you're not taking advantage of the platform. The idea behind using AWS is not just to avoid your own capital infrastructure, it's also to allow someone else to do the "undifferentiated heavy lifting". In our case:

Consul -> ParameterStore

Vault -> KMS/Cognito/IAM

Nomad -> Elastic Container Services/Fargate (serverless Docker)/Lambda

SQL Server -> RDS -> Aurora

Memcached -> ElastiCache (Memcached protocol)

Consul+Fabio (service discovery/load balancing) -> Elastic Load Balancers/Route 53/Autoscaling

VSTS (MS's hosted CI/CD platform) -> AWS Code Build/Code Deploy/Code Pipeline

"not marrying yourself to a one cloud provider" is equivalent to "coding using a repository pattern to access the database" in both cases people claim to not wanting to tie themselves to one vendor and then they realize that they aren't taking advantage of what the vendor has to offer.

In my experience, no one wants to go through the pain of moving to another vendor just to save a little money.

Netflix doesn't pay retail for AWS or G Cloud

The cloud doesn't really have economies of scale unless your workloads scale opposite of pricing - which they very rarely do. For small shops, sure, but for the example of 500 instances running - you could lease space and grow with physical servers at a better economic scale. If you only need 500 servers for 1 busy season and can drop back down to 100 then stay in the cloud. If you need 500 always running instances and your growth is 20% year over year, i can almost guarantee a good colo will be cheaper and faster if you use the same automation and configuration as code philosophies in private dc as you would in public cloud. Puppet/foreman/cobbler/ansible/chef all work great in private DC

This is probably the nicest answer. With things like kubernetes, hardware is quite abstracted. If it makes sense to use your own hardware, your software should be good enough to effortlessly shift some load to your own hardware and back incase own hardware can't handle a spike. Hybrids are probably the future where you rent when it makes sense, and buy when it makes sense and move compute and storage by dragging a slider.

Third hand knowledge - since I was not a part of the negotiations - fair warning. The company I worked for was getting discounts in the 40-50% range. It was the only way they could afford to use AWS, frankly, since it was all static provisioning (i.e. no elasticity in the workload). They were also approaching a petabyte of data stored in S3.