Hacker News Reader

Comment by srdev | Hacker News Reader

Can you elaborate on what the bin packing problem is?

Kubernetes requires machines big enough to run all your containers. Those machines are the bins. Your containers are the packages. Fitting your containers in such that there is no criticality overlap (in AWS, that all instances of service X are spread across machines in different AZs) and that there is room for immediate scaling/emergency fault recovery (headroom on the machines running your containers) gets expensive. You're buying big and running little, and that comes with costs.

Meanwhile, in AWS, you already have pre-sized blobs of RAM and compute. They're called EC2 instances. And then AWS pays the cost of the extra inventory, not you. (To forestall the usual, "overhead" of a Linux OS is like fifty megs these days, it's not something I'd worry about--most of the folks I know who have gone down the container-fleet road have bins that are consistently around 20% empty, and that does add up.)

You may be the one percent of companies for whom immediate rollout, rather than 200-second rollout, is important, and for those companies a solution like Kubernetes or Mesos can make a lot of sense. Most aren't, and I think that they would be better served, in most cases, with a CloudFormation template, an autoscaling group with a cloud-init script to launch one container (if not chef-zero or whatever, that's my go-to but I'm also a devops guy by trade), and a Route 53 record.

You're basically paying overhead for the privilege of `kubectl` that, personally, I don't think is really that useful in a cloud environment. (I think it makes a lot of sense on-prem, where you've already bought the hardware and the alternatives are something like vSphere or the ongoing tire fire that is OpenStack.)

I know you're answering the question of bin-packing, but after two years of experience with it, I can say that for me, bin-packing is one of the smallest benefits (though it sells very well with management), though perhaps a baseline requirement these days. The real benefits, in my experience, stem from the declarative nature of cluster management, and the automation of doing the sensible thing to enact changes to that declarative desired state.

Sure. CloudFormation exists for that, though, and both its difficulty and its complexity are way overstated while also letting you manage AWS resources on top of that.

And it doesn't cost anything to use.

Eh, there are a lot of terrible things I'd rather put myself through than writing another CloudFormation template for any sort of complex infrastructure. It could have been made easier and more readable if my company had allowed the use of something like Monsanto's generator [1], but creating ASTs in JSON is not my idea of a good user experience.

[1] https://github.com/MonsantoCo/cloudformation-template-genera...

I maintain auster[1] and am a contributor to cfer[2] for exactly that purpose. ;) CloudFormation really isn't a rough time anymore, IMO.

[1] - https://github.com/eropple/auster

[2] - https://github.com/seanedwards/cfer

If you know those tools exist, maybe. I just put together a new project using cloudformation (technically serverless, but it turned into 90 percent cloudformation syntax anyways), and it was pretty rough.

Maybe it's just me, but as a programmer the first thing I ever asked when looking at the wall of CloudFormation JSON was "so how do we make this not suck?".

Our job is not just to automate servers, it's to automate processes, including stupid developer-facing ones.

True, but as a _programmer_, working on a _new to me platform or package_, I am _very_ reluctant to add an extra third-party abstraction layer which requires it's own evaluation of quality and stability and some learning curve. It's gotta be pretty clear to me that it really is "what everyone else is doing", or I've gotta get more experience with the underlying thing to be able to judge for myself better.

I've definitely been burned many times by adding an extra tool or layer meant to make things easier, that ends up not, for all manner of reasons. I think we all have.

Worth noting that "nearly 31,000 AWS CloudFormation stacks were created for Prime Day" [1], so Amazon uses CloudFormation heavily internally. Not a guarantee that it's what 'everyone else is doing', but it's a good indicator of quality/stability and that it will remain a core service within the AWS ecosystem for some time.

[1] https://aws.amazon.com/blogs/aws/prime-day-2017-powered-by-a...

I think they're talking about a CF template generator being the third-party software (I could be wrong).

You're not wrong. But in my case, that had been basically forbidden as an option, essentially because it "wasn't supported by Amazon," and because there's just additional risk to non-standard approaches. AWS certifications cover CloudFormation, so you can hire for that with low risk pretty easily. Other nonstandard utilities, not so much.

Cloudformation templates can be written in YAML now, which is a lot less sucky than writing JSON by hand.

If your only experience with CloudFormation is hand-written JSON, it's worth another look.

We used to use troposphere, a Python library for generating CloudFormation templates, but have since switched back to vanilla CloudFormation templates now that they added support for YAML. We're finding it's much nicer to read and write in plain YAML. We're also now using Sceptre for some advanced use cases (templatizing the templates, and fancier deployment automation).

> If your only experience with CloudFormation is hand-written JSON, it's worth another look.

Strongly agree.

YAML and sensible formatting conventions really do transform the usability of CloudFormation.

And so does terraform which is pretty awesome!

Terraform requires significant infrastructure to get the same state management and implicit on-device access that CloudFormation's metadata service does. A common pattern in systems I oversee or consult on is to use CloudFormation's metadata service (which is not the EC2 metadata service, to be clear) to feed Ansible facts or chef-zero attributes in order to have bootstrapped systems that do not rely upon having a Tower or Chef Server in my environment.

The Terraform domain spec is not sufficiently expressive (just look at the circumlocutions you need to not create something in one environment versus another). It's way too hard to build large modules, but the lack of decent scoping makes assembling many small modules difficult too. Worse, the domain spec is also requires HCL, which is awful, or JSON, which is a return to the same problem that cfer solves for CloudFormation. One of my first attempts at a nontrivial open-source project was Terraframe[1], a Ruby DSL for Terraform; I abandoned it out of frustration when it became evident that Terraform's JSON parsing was untested, broken, and unusable in practice. Out of that frustration grew my own early CloudFormation prototypes, which my friend Sean did better with cfer.

If you're looking for an alternative to CloudFormation, I generally recommend BOSH[2], as it solves problems without introducing new ones. Saying the same for Terraform is a stretch.

[1] - https://github.com/eropple/terraframe

[2] - https://github.com/cloudfoundry/bosh

Cloudformation is not without its problems, even still. I feel you overstate Terraform's issues, though there is tons of valid criticism to go around. I would say Terraform really shines in areas CF still does not.

We use Terraform for our foundation and a clever Cloudformation custom resource hack to export values out to use in Cloudformation stacks where appropriate(use with Serverless services, etc). Works great for us; Terraform has seen significant(surprising even if you haven't looked at it in 6+ months) development over the past year.

Immutability, in other words.

It's multi-machine scheduling, basically. Given N resources and M consumers, how can I fit all M consumers most efficiently, while using the minimum N?

The bin metaphor is that you imagine one or several bins on the floor, and a bunch of things to place in bins. Binpacking is just playing tetris to make sure all your things are packed into as few bins as possible, because bins cost money.

https://en.wikipedia.org/wiki/Bin_packing_problem

If you have a bunch of jobs and you need to run them efficiently on a bunch of compute, you need to be careful not to oversubscribe the hardware, especially wrt memory. There's an isomorphism between running differently sized jobs concurrently on a compute resources, and the bin packing problem. It's a scheduling problem.

Running load efficiently on a given resource. Most VMs running a single app are under-utilized so it's more efficient to pack apps into containers and run them across a smaller pool of servers so that they all get the necessary resources without waste.

Kubernetes does really well with this, although ease of deployment using config files and the abstraction from underlying vms/servers is probably more useful for most companies.

Kubernetes emphatically does not do better at resource utilization than not using Kubernetes. You should figure on between ten and twenty percent of wastage per k8s node, plus the costs of your management servers, in a safely provisioned environment.

You can argue about the configuration-based deployment being worth it--I disagree, because, frankly, Chef Zero is just not that complicated--but it's more expensive in every use case I have seem in the wild (barring ones where instances were unwisely provisioned in the first place).

Based on what evidence? We can put hundreds of customer apps into a few servers and have them deployed and updated easily. We could try to manage this ourselves but it's much less efficient while costing much more effort. GKE also costs nothing for a master and there is no overhead.

K8S/docker also makes it easy to avoid all the systemd/init issues and just use a standard interface with declarative configs and fast deployments that are automatically and dynamically managed while nodes come and go. We have preemptible instances with fast local storage and cheap pricing that maintain 100s of apps. K8S also easily manages any clustered software, regardless of native capabilities, along with easy networking.

Why would I use chef for that - if it can even do all that in the first place?

> GKE also costs nothing for a master and there is no overhead.

Just a historical perspective: GCE used to charge a flat fee for K8s masters after 6 nodes. After the announcement of the Azure K8s service, with no master-fee, GCE has dropped the fee as well :)

Reply on news.ycombinator.com