Hacker News Reader

Comment by peterwwillis | Hacker News Reader

If you weren't running in the cloud, you would just use baremetal machines with prioritized job queues. You can keep a low-priority job queue doing video encoding 24/7, and keep any other jobs or applications at a higher-priority. The result is continuous resource utilization at near-peak with little contention.

This reduces complexity and overhead significantly. With a cloud model, each level of abstraction requires new methods for networking, orchestration, configuration, execution, debugging, etc. On a single baremetal machine, all of the resources are essentially the same things running on one layer. But you can still get fancy with cgroups, namespaces and quotas if you need enforced isolation or resource limits.

View on HN

Plus the capex costs of constantly having to buy new hardware and stick it in a datacenter for when your workload went up.

Plus the costs of maintaining that hardware both from an opex and capex point of view.

Plus the capex and opex costs of creating and maintaining the high speed network required to ship those files back and forth within the datacenter space that you're renting.

Plus the massive hit in how nimble your team can be. (Want to support a new codec? Either buy and install new hardware, or set the priority on these re-encodes lower and tell your boss to wait six months)

When you have an extremely variable workload such as Netflix et al, even though AWS/GCP/Azure are more expensive on a unit cost, your savings from operational expenditure will more than make up for that difference. Not having to buy and maintain a bunch of "just using baremetal machines" is a massive, massive cost savings.

Lambda / Google Cloud Functions / Azure Functions are a natural extension of this-- your write/test/deploy cycle can be much, much faster and you don't have to even worry about maintaining any infrastructure. The brainpower savings of this more than make up for the extra cost for CPU cycles.

Well first of all, you can always pay someone else for baremetal server hosting and support, just like with Amazon. So you don't need to buy and maintain, and you can still save costs - you're renting, just like with Amazon.

Second, even if Netflix's "workload" is variable, they're still using reserved instances. There's no practical difference from those to baremetal machines.

Finally, I am highly skeptical of anyone telling me that five more layers of abstraction make it easier or quicker to write, test and deploy code. Code is code. Servers are servers. Microservices are microservices. Different people are (or should be) maintaining all of these, and if they do their jobs right it should not incur a performance penalty on anyone else.

I will also state the obvious: if you don't hire the right people to manage the right pieces, you will waste huge amounts of time and money trying to figure out what other people have known for decades. This is universal, and has nothing to do with what technology you use.

> 300,000 CPUs

You better have a bloody big datacentre.

What happens if you need to encode a lot of videos during peak streaming?

The beauty of the cloud is that when that happens, they can burst up and use on-demand instances to increase the overall pool. With bare metal you either have to keep a bunch of spares or not be able to handle that situation.

They would turn down the amount of encoding they do during peak streaming. They use their existing servers instead of paying for more servers to encode their Netflix Originals, but encoding isn't time sensitive so can just run on a "spot instance" when their reserved instances are under-utilised.

They'll need 4K, 1080p, 720p etc encoded versions ready for the release of the series, but they might have copies of some of the episodes ready for encoding a few months before the release, so can slowly encode them on their spare resources.

I worked there so I'm well aware of how it works. My point was sometimes they need to do a massive encode during peak, and having the cloud allows them to do that when necessary.

I agree 100%. The cloud is basically the only way to do massive parallel processing in a very short time window without the up front costs. But if this event is uncommon, you can do your massive encode on the cloud, and shut off the cloud resources after peak and go back to bare metal.

I think bare metal is simpler, and that simpler often means better, but there are limits to bare metal. Those limits are where the cloud can come in. You can (and many people do) use both.

Reply on news.ycombinator.com