A Decade of Container Control at Google
nextplatform.com
nextplatform.com
I was following the container automation area very closely a couple of years ago, including some personal email interactions with Wilkes, and the primary LXC/cgroup userspace/kernelspace authors (none of whom worked at Google). It is my distinct impression, as others have noted, that Kubernetes was not an in-house for in-house thing, but rather a product that was made after many successive in-house systems specifically with a view towards sharing with the public, probably partly in response to public perceptions and popularization around LXC/Docker, and Amazon EC2's rapid success which Google's management would probably like to replicate.
Bad journalism.
Around about the same time (~2005?) SGI got cpusets merged into the kernel; this was initially just intended for pinning groups of processes on to specific NUMA nodes on big-iron systems. At the suggestion of akpm we started using it internally at Google to do coarse-grained CPU and memory isolation, by making use of the fake-NUMA emulation support to split the memory on our servers into chunks of ~128MB each and pinning each job to some number of fake nodes. This worked surprisingly well, but required painfully-complex userspace support to keep track of memory usage of each job, and juggle memory node assignments (particularly since we wanted to be able to overcommit machines, so we had to dynamically shift nodes around from low-priority jobs to high-priority jobs in response to demand).
The cpuset API and abstractions turned out to fit the resource control problem pretty well, and they had already been merged into the kernel, which gave that API a kind of pre-approval compared to the other generic resource control approaches. So we worked on separating out the core process/group management code from cpusets, and adapting it to support multiple different subsystems, and multiple parallel hierarchies of groups. The original cpusets became just one subsystem that could be attached to cgroups (others included memory, CPU cycles, disk I/O slots, available TCP ports, etc). It turned out that this was an approach that everyone (different groups of resource-control enthusiasts, as well as Linux core maintainers) could get behind, and as a result Linux acquired a general-purpose resource control abstraction, and other folks (including some at Google) went to town on providing mechanisms for controlling specific resources.
The namespace work was going on pretty much in parallel with this - it wasn't something that we were interested in since it was just added overhead from our point of view. The jobs we were running were fully aware that they were running in a shared environment (and mostly included a lot of Google core libraries that made dealing with the shared environment pretty straightforward) so we didn't need to give the impression that the job had a machine to itself. IP isolation would have been somewhat useful (and I think was later added in Kubernetes) but wasn't very practical to provide efficiently given Google's networking infrastructure at the time.
We weren't really interested in LXC since we had our own userspace components that had developed organically with our container support (and which as others have commented were so entwined with other bits of Google infrastructure that open-sourcing them wouldn't have been practical or very useful).
That said, we had been discussing a Borg-as-a-Service for quite some time BEFORE Docker became popular. Docker's popularity was undeniably a catalyst in getting Kubernetes built.
Refers to this post of three weeks ago ~ https://news.ycombinator.com/item?id=11216020
If they can commoditize the platform from underneath Amazon, they believe they can build out better/faster/cheaper cloud infrastructure.
To be clear, this is exactly why we donated 100% of Kubernetes to the Cloud Native Computing Foundation. We believe very strongly that the reason Linux was successful was that no single company controlled it. Though we have opinions about what makes container management successful, we are but a minority of the more than 650+ contributors to the platform.
1 - You don't get to share the burden of the maintenance with other users of the code.
2 - It's guaranteed that after every acquisition the code-base of the bought company will be hard to integrate; it's been written against the publicly available infrastructure, which is of course wildly incompatible with the internal one.
3 - New hires will often already know how the open source tools work, while they'll have to learn all of the proprietary equivalents from scratch.
4 - Engineers will generally prefer to work on projects where the knowledge is transferable elsewhere. Staffing is easier if you can say with a straight face that people will not just be working with a big proprietary ball of mud, but at least partially with publicly available technology.
The other reason I can think of is that open sourcing parts of existing code bases is a lot harder than it looks [0]. It gets harder if the code is in a monorepo [1]. It gets harder if there's a strong culture of library reuse. The further you get away from the center of the dependency graph, the harder it gets. Could you open source Borg without open sourcing Chubby? GFS? Babysitter? (My recollection, which might be faulty, is that even long after Babysitter was otherwise dead, it was still used for bootstrapping Borg clusters).
And since the goal has to be to create a viable project rather than just a code dump, there has to be a way for outside contributors to make changes. That needs to be possible for both the project itself, and all of its dependencies. Either you need realtime synchronization between the internal and public repos, or you have to make the public one the repository of record, and pretty much give up on the concept of the monorepo. Which would have been a really hard sell at least at last decade's Google.
It's a lot easier to do this for green-field projects, with plans for releasing as open source being taken into account right from the start.
[0] I wrote more on that subject a year ago; https://www.snellman.net/blog/archive/2015-03-19-cant-even-t...
Kubernetes: our container services are best; run your containers here, but if you don't want to anymore, run them anywhere using exactly the same api.
gRPC: our pub/sub is the best; run your rpc here, but if you don't want to anymore, run them anywhere using the same api.
tensorflow: our managed learning api is the best; run your learning here, but if you don't want to anymore, run it anywhere with the same api.
etc etc with dataflow and apache beam. These are best in class services, and except for bigquery (which has been a big driver of some of the big-name moves recently), they're using this model to take away the risk of transition.
It's pretty smart.
So what's the real reason nobody gets to see this code?
Thank you for the details!
We made a very strategic decision to rebuild it. It embodies many lessons from Borg and Omega (both things we got right and things we botched). It is implemented in an easier-to-approach language (Go) which has an active OSS community. It specifically focuses on "legacy apps" (everything written up to and including today) and open standards (HTTP, JSON, REST).
I've never been shy about my opinion that I do hope to supplant Borg one day, but that day is necessarily years away. Of course no major project has ever used it, the whole thing didn't exist just 2 years ago.
Rubbish, and saying so undermines the rest of your points.
> Rubbish, and saying so undermines the rest of your points.
Hardly rubbish. Although there are open-source projects which use C++, I and many others avoid them like the plague.
I think he meant 'approximately ZERO' in the Spolsky sense, which is 'sure, there are some, but in the grand scheme of things they're indistinguishable from ε.
The Go community is vibrant and growing. Go is an easy language to learn (and I say that as someone who LIKES the power of C++) and it is not a total joke to ask people who report bugs to jump in and try to fix them. C++ is simply NOT approachable by mere mortals, and would have made for a very different community and a much slower pace.
And I say that as someone who detests many facets of Go - but it's just better at Getting Things Done than C++.
Much of the organic growth, as you put it, can also be described another way: as an accumulation of useful features and optimisations.
The language issue was addressed by another response. To claim there's no open source C++ community which is why it's written in Go is just bizarre. There's absolutely a thriving open source C++ community, but if having the biggest open source community was the driving factor in picking the implementation language then I guess you should have picked Java.
Truth is, a LOT of people don't write code. They write content and use pre-built code (think WordPress). Borg simply can not accomodate that very well. It's simple as long as you control things from soup to nuts.
Yes, some "organic growth" was useful features. But a lot of it was useless features, or features that are now obsolete but can't be removed because someone somewhere is using them, and probably doesn't have enough time to re-test without the feature (true story).
I don't think Borg imposes all that many requirements on jobs, really, and the few it does can be disabled. Or at least could.
But I guess we're probably wandering out of the area covered by the papers now.
The first and best is to make all your MySQL IO go to an external cluster filesystem or other remote IO system. Because MySQL supports pluggable storage, you could write an Hadoop FS storage manager. This has the advantage that if a single MySQL instance is blown away, all the committed data is available for a new replica to start reading. I don't know if Docker or other container systems support automagically turning local IO calls into remote IO calls (or whether that really makes sense in a MySQL environment), but that's a similar approach. Condor supported this through their remote libc interface.
The second is to use some sort of per-task persistent local storage. In Docker world, this would be a mounted volume- the docker host would manage the storage, and new containers would remount that storage. You could have a process that restored the local storage from a backup, and the use replication from a master to catch up.
The third would be to have some sort of per-container persistent storage (the Borg paper calls this an "alloc").
For the server, most people wouldn't have Apache bind port 80 inside the container- you'd bind another port, and use some sort of other mechanism such as load balancing to expose the web server on a standard port
I also don't blame people for being confused about Google's container infrastructure. Google has issued blog posts in the past that were misleading (at best) about the relationship between Omega, Borg, and Kubernetes.
A decade or history lead to the knowledge that a particular style of monitoring was needed. That knowledge lead to cAdvisor. Is it perfect, of course not, but it fills a need and is directly derived from Google's experience. I fail to see how that misattribution, personally.
Tim knows more about this than almost anyone, but I will add one point - we have used Kubernetes for significant internal projects, and plan to continue expanding its usage over time.
To Tim's point, though, it'll take time. The thought that you could move literally millions of lines of code and applications over to a new platform in just 10 months (the amount of time that Kubernetes has been GA) is... optimistic.
This is NOT in competition with Borg, though. Not yet.
Capacity planning and actual design of a system is more important for most companies.
I have seen several companies use 3 to 4 times the hardware it needs because they want to do containers or the private cloud thing.
so....
1 VM to run 1 docker container to run 1 jvm .... probably 10 vms per host.... instead of 20 jvms per host
...and we added consul + Haproxy and a few other pieces we didnt need ( think srv records ) ... which added yet more hardware
That doesn't make sense, the Linux overhead is not enough to make it that much less dense. You get better HA by running 20 VM's instead of 20 JVM's because machines crash taking out 20 apps instead of 1. I know the physical host can crash taking out 20 hosts, but you have that problem in either scenario.
2) Is it even worth it or necessary as afaik, even the biggest of enterprises' infrastructure needs are considerably less than those of internet companies?