Why Kubernetes is winning the container war
infoworld.com
infoworld.com
Kube is winning for the same reason React/Redux (and now Mobx) is winning and why Rails was winning at the time. Community.
The community for Kube is awesome and the work people are doing in the field is being noticed all over the place.
I've seen people (myself included) that moved production clusters from mesos to Kube just because the activity of the development and how secure they feel with the community and the project going forward.
React and Rails (at the time) had the same sort of pull towards the community and why a lot of people on-boarded.
Golang is most likely a factor here too. I feel most people find Golang friendlier than Scala/Java. That's why Kube has many more contributors, the hurdle for contributing is easier to jump
Sorry I am new to this. I saw a presentation where someone was running Kubernetes on DC/OS . Why are people doing that if they can just run pure kubernetes.
for example, if you restart a Kafka node on any container orchestration platform, container is destroyed and respawn on another node which is great, but you lose all your data and the broker has to rebuild all that queue again, while Kafka on DC/OS wait and maintains the volume just in case you are having temporary maintenance restart before relocating the workload.
another example is Cassandra if you want to add more than 1 node, the traffic to resync the data across 2 nodes can actually degrade the performance on your cluster or even taking it down.
I can go on and on, all of these issues and requirements come from running these mission critical data apps in production, with each application having its own logic on how it handles day 2 operations, but you don't know what you don't know unless you tried running them in production.
Some people talk about the custom scheduler feature or Petsets in K8s, those features are cool and hats off to the K8s community, but they are still early in development while the framework architecture is a core mesos functionality since day 1.
And finally, everyone is free to run whatever they want/like, each company and person might have different preferences on how they want to run workloads and how much effort they want to go through that, I don't think it's productive to go to another VI vs emacs here ;-)
Since I have set-up both in production (and more than once) I can tell you for a fact. Kube is easier to setup and easier to run with less problems along the way.
Since I am running essentially the same cluster on kube that I ran on mesos, this is a no-bullshit comparison.
[To be fair] I never used dc/os, I used marathon and chronos on top of mesos.
I am working on an open source project called "The Startup Stack" to essentially help startups bootstrap (and manage) a cluster in the cloud. link: http://docs.the-startup-stack.com/
Right now, it's working using mesos/marathon etc... 2 companies are using it in production.
In the last couple of months I've been thinking about making the move to Kube for this project. I might evaluate DC/OS as part of it.
Working on this project made me appreciate how hard it is to generalize and hit everyone's taste. There's a lot of customizing around cloud infrastructure now and I definitely feel there are more common grounds we can create.
I'll give DC/OS a shot before making my final decision.
Thank you for your suggestion and your awesome comments on this thread. I was always impressed with the mesosphere team in previous discussions here on HN.
Coincidentally, you announced the open source version on my birthday this year. Maybe it's a sign :)
I went to MesosCon this year and the previous. Last year it was a really fun kind of scrappy conference. This year it was basically "buy DC/OS". Even the open source vendor booths all had "works with DC/OS" stickers. It reminded me of "Dell recommends Microsoft Windows XP Professional for the best experience" crap that Microsoft used to pull with OEM's licensing windows. I was really unimpressed. Whereas Mesosphere is adding hooks into mesos to add things like AAA (Authentication, Auditing, Authorization) and then implementing it as proprietary modules in their commercial DC/OS, It is native and open source in Kubernetes (thanks to Redhat contributing code from openshift).
So on one hand I see a young ecosystem being swallowed by a single vendor who is slowly squashing the community. On the other hand I see several vendors doing everything in the open and a huge community. With the technical capabilities of both quickly converging as k8s implements a lot of the things Mesos used to have over it, Mesos is becoming less and less "sexy" and more like a liability unless you want to roll your own everything like Apple did with their Jarvis scheduler for Siri.
This is a bit disappointing and disconcerting to hear! I understand that technology doesn't happen in a vacuum, but when you have two well supported/known technologies, picking the most 'popular' one rather than something based on technical merit is a huge issue for me.
Could you define the "everything" please.
What applications are you running? How many of them? How much resources do they use?
For example, the PetSet feature of Kubernetes 1.3 involved a really long discussion with people trying to work out how and why to use it.
I remember back in the Kubernetes 1.1 days when I was trying to get AWS ELBs to talk with Kubernetes and possibly finding drivers to automatically set up proxy points. In my search, I found the Github issues where people were discussing that very thing. The person who wanted AWS integration was there talking about the specific needs and quirks of the AWS ecosystem. There were a series of patches made to beef it up, and the discussion on Github documented how the feature was conceptualized (so as to be part of a more general solution on port management).
In contrast, I don't see that with Docker. I can't say anything about Mesos, as I've never tried using it.
Community goes a long way but has to co-exist with strong technology and being open to change.
The k8s team is very responsive and I'm sure these will be ironed out in the near future so we can all adore each other's cattle :)
Otherwise you need to separate your available container hosts into clusters: Elasticsearch cluster, Cassandra cluster, etc. and treat those differently from your machines you deploy your other apps to, which to be fair, they are different and need to be treated differently.
We tried:
* gluterfs - cluster can be setup in seconds, really. Just launch daemon sets and manually (but you can automate this) create a cluster, but we hit to that fact that CoreOS can't mount glusterfs shares at all. We tried to mount NFS and then hit next problem.
* NFS from k8s are not working at all, mostly this is because kubelet (k8s agent) need to be run directly on a machine and not via rkt/docker. Instead of updating all our nodes we mounted NFS share directly to our nodes.
* PostgreSQL we haven't tried yet, but if occasional pod kill will take place and then resyncing database can became huge issue. We ended up in running pods that is dedicated to specific node and doing manual master-slave configuration. We are not tried other solutions yet, but they also questionable in k8s cluster.
* RabbitMQ - biggest nightmare of all of them. It needs to have good DNS names for each node and here we have huge problems on k8s side: we don't have static host names at all. Documentation said that it can, but it doesn't. You can open kube-dns code it doesn't have any code at all. For pods we have only domain name that ip-like: "10-0-0-10". We ended up with not clustering rabbitmq at all. This is not very important dataset for us and can be easily lost.
* Consul - while working around problems with RabbitMQ in k8s and fighting DNS we found that Consul DNS api works much better than built-in kube-dns. So we installed it and our cluster just goes down when we kill some Consul pods as they changed it's host names and ip. And there are no straightforward way to fix IP or hostnames (they are not working at all, only ip-like that can easily changed on pod deletion).
So best way is to have some fast(!) external storage and mount it via network to your pods, this is much much slower than direct access to Node's SSD but it give you flexibility.
If you set it up in HA mode, then in theory you don't need persistent volumes, although RabbitMQ is of course flaky for other reasons unrelated to Kubernetes -- I wouldn't run it if I didn't have existing apps that relies on it.
I'm surprised because I know teams which are very satisfied with running RabbitMQ at scale. Could you elaborate?
With HA mode enabled, it will behave decently during a network partition (which can be caused by non-network-related things: high CPU, for example), but there is no way to safely recover without losing messages. (Note: The frame size issue I mention in that comment has been fixed in one of the latest versions.)
We have also encountered multiple bugs where RabbitMQ will get into a bad state that requires manual recovery. For example, it will suddenly lose all the queue bindings. Or queues will go missing. In several cases the RabbitMQ authors have given me a code snippet to run with the Erlang RELP to fix some internal state table; however, even if you know Erlang, you have to know the deep internals of RabbitMQ in order to think up such a code snippet. There have been a couple of completely unrecoverable incidents where I've simply ended up taking down RabbitMQ, deleted its Mnesia database, and started up a new cluster again. Fortunately, we use RabbitMQ in a way that allows us to do that.
The bugs have been getting fewer over the years, but they're not altogether gone. It's a shame, since RabbitMQ should have a model showcase for Erlang's tremendous support for distribution and fault-tolerance. You're lucky if you've not had any issues with it; personally, I would move away from RabbitMQ in a heartbeat, if we had the resources to rewrite a whole bunch of apps. We've started using NATS for some things where persistence isn't needed, and might look at Kafka for some other applications.
I prefer to have a DB which is "cluster-aware", in this case, you can tag your hosts/Nodes and use Node affinity to scale your DB service so that it matches the number of Nodes which are tagged - Then you can just use the a hostDir directory as your volume (so the data will be stored directly on your tagged hosts/Nodes) - This ensures that dead DB Pods will be respawned on the same hosts (and be able to pickup the hostDir [with all the data for the shard/replica] from the previous Pod which had died).
If your DB engine is cluster-aware, it should be able to reshard itself when you scale up or down.
I don't think it's possible for a DB to not be cluster-aware anyway - Since each DB has a different strategy for scaling up and down.
Basically the solution (until we get PetSets at least) is to:
1. manually spin up two pods (gluster-ceonts image) without replication controllers because if they get down we need them to stay down so we can manually fix the issue and bring it back up.
2. each pod should have a custom gcePersistentDisk mounted under /mnt/brick1
3. from within each pod, probe the other (gluster peer probe x.x.x.x)
4. each pod should preferably be deployed to it's own node via nodeSelector
5. once the pods have been paired on gluster, create and start a volume
6. make a service that selects the two pods (via some label you need to put on both)
Your wish is my command! https://kubernetesbootcamp.github.io/kubernetes-bootcamp/
Suggestions where I can find internals documentation/book? Other that the obligatory 'look in the source code'.
These days, with 3-6 month release cycles, I can't see how a traditional publisher could stay current unless you have a rapid iteration cycle.
Also, I wonder how platform-agnostic the book is? I know many resources out there start by having you set up a GCP account.
This book does the same re GCP with most stuff done via the gcloud CLI that I've seen anyhow(I've only skimmed the book). Pretty sure Kelsey Hightower works for Google so it makes sense it is specific to their products.
Kubernetes Developer docs: https://github.com/kubernetes/kubernetes/tree/master/docs/de...
Kubernetes Developer proposals: https://github.com/kubernetes/kubernetes/tree/master/docs/pr...
They're by no means perfect, but we try to be super open about our development process.
Many of the github issues also have comprehensive discussions of topics, searching the repo for topics you're interested in will often surface interesting discussions.
If you want to get more deeply involved, I'd suggest the Kubernetes community meeting (Thurs, 10am Pacific)
I learned a lot from Kelsey's "Taming microservices with CoreOS and Kubernetes" at OSCON EU 2015 https://www.safaribooksonline.com/library/view/oscon-amsterd... and https://github.com/kelseyhightower/intro-to-kubernetes-works...
It gave me invaluable in-depth understanding, regretfully a little outdated by now.
EDIT: I see that his book, due soon, has the same information as in the presentation, and much more.
We have total respect for K8s, but I don't think you can claim winning just based on the community and stars.
OpenStack has a huge larger community of developers and advocates, but it is still haven't reached it's potential despite many years and incredible effort, seminars and summits
Also most of these next gen infr projects now (DC/OS which is powered by Mesos, K8s, docker swarm ) are converging feature wise, but they also have their strengths and weaknesses, some of these are temporary, some of these are structurally by design
Mesos (and DC/OS) were not just designed for scale, but also for extensibility and different workloads, which is why you can run Cassandra, Kafka, Spark, ..etc In production. None of these workloads run as traditional containers, with DC/OS they have their own operational logic such as honoring placement, simple installation, and upgrade, which is a core design function of the two-level scheduler
People usually complain that standing up Mesos is hard, which is why we we built DC/OS and open sourced it to give you the power of mesos and its awesome capabilities in managing containers and data workloads without going through all the effort to stitch everything yourself. Check it out at DCOS.io, I am sure you guys will be blown away.
Mesos provide the core resource management and scheduling, think of it as the Kernel of an (DC) operating system. On top of Mesos you usually need:
1- Container/Workload Scheduling capability which is Marathon, now built in and integrated in DC/OS so you are just up and running in minutes 2- Management (Beautiful DC/OS UI and CLI) to manage all cluster and workload operations. It also includes Installer (of all components including Zookeeper, networking, ..etc) 3- App Store (Universe) that allows you to easily find and deploy latest packages such as Kafka, Cassandra and Spark with a single command, in regular Mesos you needed to find the Framework that worked with a specific version of Mesos, troubleshooting, ...etc. Now it's an app store like experience 4- Core infrastructure components such as very cool service discovery and loadbalancing (Minuteman), Storage drivers, CNI support
If you want to run docker containers, DC/OS also gives you the ability to use the docker daemon or the new Universal containerize (technology preview) which we just released.
Because mesos is very modular and very powerful, it was built to have modular implementation of all system components, but our team and our founders ran some of the largest implementations of Mesos (Twitter, Airbnb, ..etc) and they complied all thes best practices and the bits and pieces in DC/OS.
Hope that explains,
https://github.com/dcos/lashup
Just wanted to say that is some seriously good stuff -- a CRDT based, distributed masterless database, and what seems like a failure detector as well.
- https://github.com/dcos/dcos-vagrant
- https://github.com/dcos/dcos-docker
However, your point is somewhat valid in that Mesos requires you to allocate resources to everything you run, while resource allocation/enforcement is optional in Kubernetes. So you can easily run too many things and freeze your computer with Kubernetes, while Mesos is more conservative.
I played with Mesos & k8s and I picked Nomad instead. Now I'm not managing a huge fleet of servers I want to abstract away as much as I wanted a simple, consistent framework for allocating resources to containerized tasks across a small group of nodes. And I don't think that use case is anything to sneeze at and for a new user there just isn't anything out there as easy as nomad IMO.
The cool thing is that you can set up and environment using cattle (their own orchestration system), k8s, mesos or swarm. Up to you and your preferences.
1) Use one of Chef/Puppet/Salt/Ansible to orchestrate
2) One if those in item 1 will use Vagrant which
3) Uses Docker or Kubernetes to
4) Set up a container which will
5) finally run the app
Chef/Puppet/Salt/Ansible these days are just used to provision 1 VM/container (if even that, many people have just reverted to shell scripts) and that container is then managed using a container management tool (Swarm/Kubernetes).
Isn't Ansible basically just a way of executing shell scripts across multiple hosts in a sensible manner? At least that's how it's been explained to me.
Elastic Search 'beats' ( https://www.elastic.co/products/beats ) can do all the monitoring/stats, deploying snapshots to multiple servers can also be automated in Go or drop a Go binary on the server for some kind of Command&Control architecture for continuous remote maint, starting/stopping containers ect (assuming all security precautions have been considered).
This worked for a deployment of ~75 docker containers ymmv
Host monitoring you would utilize Kubernetes daemonsets which install whatever monitoring tool you want to use onto every single node in a cluster.
It's also the part that people are generally trying to address by using containers
1) Developer pushes code to repo, its tested, if pass>
1) Rkt/Docker image made by CI/CD system and pushed to docker registry
2) Automatically deploy new image to staging for tests and master for production (can be a manual step)
3) Sit back and relax because my time invested up front saves me hassle in the future
Setting up bare metal instances, even when using something like ansible is slow and cumbersome compared to what I'm now doing with k8s
Doing an apply/patch to kubernetes nodes takes seconds (assuming services aren't affected).
edit: Sorry, for the unfamiliar by services I mean load balancers/ingress, it's a k8s term. It takes 45s-1minute to update glbcs so I modify them as rarely as possible.
We push our docker images to quay.io but pull+deploy is still manual. How does the target platform detect a new container on the registry in order to kick off a pull+deploy?
[0]: https://github.com/alekssaul/webhook/tree/github-support [1]: https://github.com/alekssaul/simplewebapp/pull/40
sed -i -e 's^:latest^:'$VERSION'^' kube/myapp-deployment.yaml
kubectl apply -f kube/nodeapp-deployment.yaml --record
Edit: Also I just realized my numbers are 1, 1, 2, 3. Woops.
Drone can also run CI tests and build stuff in temporary containers, which avoids the need to compilers etc taking up space in the final image, and negates the need for squashing (which busts your cache).
Much faster than Quay. Quay starts a new VM in AWS every time it builds, which is super slow. Drone just starts containers.
(You have to self-host Drone, though. Since it requires running Docker in privileged mode it's not a good fit for running under Kubernetes, unfortunately.)
(Kubernetes on GKE always runs in privileged mode, but that doesn't mean it's a good idea!)
We have solved #2 by deploying an internal docker registry [1], push to it once a build succeeds, and automatically (or not) trigger deployments once an expected image lands in it [2].
In the future we will support webhooks so you can trigger deployments from external registries such as quay.io or DockerHub.
You can have a look at the all-in-one VM we have:
git clone https://github.com/openshift/origin
cd origin
vagrant up
vagrant ssh
oc cluster up
The last command will spin up a cluster with a docker registry, a HTTP router, and a bunch of images for you to play around with.
[1] The same underlying registry thay powers DockerHub: https://github.com/docker/distribution
[2] A Kubernetes controller loop that watches for new images and triggers new deployments.
Two to twenty hosts? Harder, especially when ensuring that the hosts are set up identically.
Twenty-one hosts on up? Very hard.
Automated creation and removal of hosts? Virtually impossible.
Steps 3 and 4 can be avoided if you like, using any number of techniques (including the good, old fashioned linux package with init files), but they do make a number of challenges disappear (such as consistency in code and deployment across hosts) when you run into those problems.
So, no, not everyone needs Kubernetes (I'd argue most people don't), but I can't see not using 1 or 2 (even if 1 is bash scripts).
Think of it like this. If you need to build one moderately simple widget, is it faster to build it by hand, or to build an entire factory to make it? Obviously, building it by hand. Now, how about if you need to build 100, or 100,000? How about if they all need to be exactly the same? Factories start to make sense at a certain point.
1) Use one of Chef/Puppet/Salt/Ansible to bootstrap docker
2) Use docker to build a container for the app (once)
3) Use docker to start your container on multiple hostsWhen you need to automate the entire deploy process, possibly scale up or down the number of servers/services running based on demand, or to run the same operation across hundreds of servers: that is when you need Kubernetes. Trying to do any of these tasks manually when you're dealing with a large number of machines/services/servers becomes unmanageable very fast.
This. Engineering excellence is secondary. You can get away with complete craptitude in your tech if you can build community. (I won't name examples.) Of course, it's better if you also have technical excellence. On the other hand, you can have technical excellence, but it will come to naught if you have community destroying anti-patterns.
For instance, with k8s, out of the box every running container in a clustered system is discoverable and has its own IP. If you're writing distributed applications, and you're using containers principally as a tool to make your life easier (and not as part of an internal paas or for handling data pipelines or some other use case), having that sort of discovery available out of the box is great.
[1] To distinguish from cattle. If you have a red blue and green goldfish, and your red goldfish dies, you can replace with another red fish and not really notice, but if it's purple, the others won't play with it.
Run haproxy in host networking mode, as a daemonset, and use it as your ingress point?
But here is a Zookeeper example: https://github.com/kubernetes/contrib/tree/master/pets/zooke...
Another option is running Nginx via the Nginx Ingress Controller, which is also great.
PetSets will probably be beta in 1.5. The problem with rushing PetSets is that the worst thing that can happen would be that we say PetSets are beta and then it eats someone's DB for lunch.
The goal is to get the core function stable and predictable (put enough stress testing and partition testing that we are comfortable with it) and put it into a longish beta while we add out the features that are necessary to make it able to predictably and stably keep running your clustered services. We want beta to mean "stable enough to trust".
FWIW, some other less-stable features (the new volume attaching logic in KCM, for example) should have been marked as beta. You guys are not quite consistent about what you feel is stable. :-)
We promise to do a better job of communicating alpha status - it's been a topic of a lot of discussion recently. To be fair - we've only had two (almost three!) update releases so we're still trying to be consistent in our messaging. 1.4 will try to be a lot better.
Chef's product is called Habitat https://www.habitat.sh/ and it has some VERY interesting concepts that if Kube will implement it will be much more interesting (to a lot of people).
Right now, the deployment and configuration of an application is supposed to be separated but I feel they need to be just a bit more coupled. An engineer that develops the application will define a few things like "domain" and how you connect to the application and this will be consumed by the scheduler.
Right now, dc/os and mesos are really fine grained around the DevOps people and I feel that the first that will crack the "batteries included" approach will win the fight by a knock-out.
Imagine something like Heroku on your own infrastructure, if an engineer can launch and deploy a micro-service with the same ease they deploy and access a Heroku application. That will be awesome.
So you have to maintain a private "release fork" that you periodically rebase into. This becomes a workflow bottleneck, and the VCS history will not look good, either. Even worse if you have multiple branches you deploy from.
Right now, 99% of people I know use Github-Workflow, master is always deployable (often CD), other branches are able to deploy using "hubot deploy branch-name to xxx"
I think if you take the latter and you make it easier to deploy an application on top of your cluster, you can gain a lot.
The workflow today is more cumbersome. If you could create your application, attach some definitions to it and launch it to the scheduler, it would be better.
You can dynamically configure kubernetes from within the cluster. You can either use client-go library or talk to the api server directly with rest. For other languages than Golang you may need to set up proxy server, but it's a single kubectl command.
So all the basic functionality is there, but IMO configuration should be separate if it is possible - having it in source control is nice. I only use dynamic api for scheduling kubernetes jobs.
The app decides what it runs and how, you can set configuration like domain, whether the application is public/private etc...
Once you push the application to the scheduler it will handle the rest.
I wrote a blog post about using go kubernetes library to generate yaml : https://kozikow.com/2016/09/02/using-go-to-autogenerate-kube.... It's quite nice, even for static config IMO, due to type safety.
You can easily write golang binary that would ask user series of questions "what database would you like", "do you want public access", etc. and get yaml as output.
I think most people choose Heroku over rolling their own infrastructure simply because of the ease-of-use for getting started.
I previously tried the Mesos/Marathon route (with Mesosphere and then again with Mantl) and that was nothing but a huge waste of time due to all the maintenance that was necessary for all the required servers. With Rancher, it's just spin up a container for each host with a single command and you're done.
It's all about knowing how to build an open source community -- plus experience running applications in Linux containers, which Google invented
From what I understand, and is completely not in the article, Mesos is designed for scale while most start-ups (and even established companies) can't afford or justify. K8s is simpler but still robust. Better than just fleet or compose and clearly still better than swarm (based on posts read here on hn).
> The setup to get k8s running isn't great,
+1 But they are greatly improving.
For a local test environment have a look at OpenShifts 'oc cluster up' https://github.com/openshift/origin/blob/master/docs/cluster...
All of the code Red Hat contributed around Kubernetes storage plugins went into Kubernetes upstream. If you have any questions or problems, feel free to raise an issue or reach out to us in the community. Red Hat has a large number of contributors on the Kubernetes project and we are big fans of the community and the technology!
Red Hat OpenShift provides a an enterprise-ready Kubernetes distribution that also builds on top of Docker, RHEL/Fedora/CentOS and includes integrated networking (OVS), routing (HAProxy), logging (ELK stack), metrics, image registry, automated image builds & deployments, integrated CI/CD (Jenkins), and self-service UX (Web, CLI, IDE). You can check out the free Origin community distro here - https://github.com/openshift/origin or sign up for our free hosted developer preview here: https://www.openshift.com/devpreview/. We also offer commercially supported solutions - https://www.openshift.com/container-platform/.
It will make setting up a cluster as easy as 'kubeadm init' on the master, and 'kubeadm join --token $TOKEN --api $IP' on the nodes.
Video demo: https://www.youtube.com/watch?v=8cirvYVhRoI&feature=youtu.be...
The commands to test it: https://gist.github.com/errordeveloper/bdd268e71fc8916c1d0ef...
And the work that we have been going on with kubeadm. https://github.com/kubernetes/kubernetes/pull/30360 https://www.youtube.com/watch?v=Bv3JmHKlA0I
- https://jujucharms.com/canonical-kubernetes/
Currently in alpha, feedback and PRs accepted, etc.
Disclaimer: I work for Canonical
Running k8s in AWS with Rancher - http://rancher.com/running-kubernetes-aws-rancher/
August 2016 meetup on HA k8s Clusters - https://www.youtube.com/watch?v=ppY9cqTvBVE
Community is a big deal, people tend to underestimate this; Putting aside newer companies, when a larger enterprise ventures out to open source they do take community as a major factor as you have to consider the tool you build will have to last 5 years, maybe 10, maybe more.
To delve further into the Docker side of things, I personally wish that the company would focus on its core business instead of stretching itself with extra things. I get the need to improve the UX, which they do very well considering how far we have come from LXC.
I feel Mesosphere starting to go down the feature creep route as well, but I wish them all the best as I loved Mesos since the beginning all those years ago.
Granted, the title is fanboyish, but it really seems to be a significant response to Kubernetes.
What we now need is tools that allow architectural fences around components, and reliability guarantees around subsystems ... versus not only technical but also business-level risks (including state-level actors)... often across borders, including for example exchange rate risk. This is based on business-level risk models, not some engineer feels X or Y type reasoning which is (often very well, but broad-picture uselessly) based on technical know-how.
I prototyped such a system pretty successfully, you can read the design @ http://stani.sh/walter/cims/ .. it's incomplete (critically hard to explain investment utility for non-tech business types) but at least infrastructure agnostic and practically informed.
NB. To be non-humble, by way of demonstration I am a guy who has been called to conceive/design/set up/manage from-scratch datacenters for extremely visible businesses with highly motivated attackers with USD$Ms at stake for penetration. Systems I personally designed run millions of USD/month and have many times that in investment. And it's obvious that both Google ("compete with amazon on cloud business... high-level desperate for non-advertising income!") and Docker ("VC! growth! growth! growth!") have their own agendas here... we should trust no-one. It's early days. Bring on the ideas, build the future. Hack, hack, hack. We the little people own the future. It's ideas.
For instance, it certainly is possible to run a Cassandra cluster by having each instance run in its own Docker container. My understanding is that it would be much more efficient to run this cluster with a dedicated Cassandra cluster instead.
Is this right? Or are the performance benefits of running a dedicated Cassandra scheduler on Mesos negligible compared to running them in containers?
Also, if you look at the main Kubernetes repo on GitHub, you can find the yaml files for various popular open source projects in there as well. Some of them are out of date though - Things have been changing so fast.
I think a problem that still comes up is that you still need yaml files to account for the interactions between different services too. Each service doesn't live in a vacuum.
We're currently working on building an complete auto-scalable stack on top of Kubernetes and a deployment service to go with it. See https://baasil.io
Everything we do is open source except the actual website.
http://kubernetes.io/docs/user-guide/volumes/#vsphere-vmdk-e...
I believe is inadequate
The documentation is lacking. Badly. And doesn't seem to keep pace with the features that are added in every point release, or in the alpha/beta versions. A lot of what we've learned has been by experimentation, poking at things, following the k8s user discussions, watching Github, reading code.
Also, have you seen the box at the bottom of the docs that says "I wish this page ..."? It goes right into their issue tracker, which increases the likelihood of something getting fixed.
I should keep notes on specifics, so I apologize that I can't highlight a particular thing that I've been frustrated by in a given moment. I'll certainly say that the documentation is improving.
As a general item, I think my biggest struggle has been hitting a wall with the documentation - there are some things that are left almost as exercises to the reader, especially getting into more advanced topics (how does one do rolling upgrades or canary deployments in more esoteric situations, how might one do load balancing properly in a master-slave style configuration versus something more round-robin oriented, etc.)
And I don't like to levy a complaint without acknowledging that anything opensource is ripe for improvement by contribution, but my professional situation prevents me from doing so. Trust that I would love nothing more than to help and not just make an empty observation or moan about it.
how does one do rolling upgrades or canary deployments in more esoteric situations
So there are a couple of moving parts to that: foremost, Kubernetes is only a tool, and so I would doubt there is The One True Way(tm) of doing canary deployments -- and I'm taking liberties with your comment in that I presume you don't mean the literal rolling-upgrade which is ``kubectl rolling-update`` and is very smart. Having said "it's just a tool," the first thing that sprung to mind when reading canary deployments was the health checker built into the ReplicationControllers. There are several ways one can check on the "health" of a Pod, and k8s will remove any Pod from service that has declared itself ineligible for service.
If the QoS metrics are too broad for any one given Pod to have insight into, k8s has a very robust API through which external actors can manipulate the state of the cluster, rolling back the upgrade if it turns out the new deploy is proving problematic.
I hope he doesn't throw tomatoes at me for suggesting this, but Kelsey Hightower <https://github.com/kelseyhightower> is an amazing, absolutely amazing, community resource. If you don't find inspiration from the talks he has given, then reach out on Twitter and ask for a good place to read up on your concerns. I bet if he doesn't know the answer, he will know the next contact for you to try.
load balancing properly in a master-slave style configuration versus something more round-robin oriented
Now there is something you and k8s may genuinely have a disagreement about - in its mental model (to the best of my knowledge) every Pod that matches the selector for a Service is a candidate to receive traffic. So in that way, there is no "slave" because everyone is equal. However, if I am hearing the master-slave question correctly, failover is automagic because of the aforementioned health check.
edit I want to also add that the Kubernetes community meeting is fantastic. I don't attend every week but I do play catchup on the videos released after.
Regarding rolling updates, I have found that looking at the kubernetes API - and discerning what kubectl is doing under the hood - is helpful in some cases. We've taken to API inspection to assist with some of the more complex update patterns we're trying to address.
Completely agreed on the value of Kelsey to the community. I hesitate about contacting any one person directly on a topic, but his guides and github repo are just outstanding.
On the load balancing/master slave thing - I would have agreed completely with you. Master/slave configurations seem antithetical to some of the core Kubernetes concepts... but then the waters are becoming muddied because it seems like PetSets are a response to that missing pattern. I think you're right on the mental model. Every pod is (from what I understand), or at least should be, seen as an equal candidate, the only thing being "can it receive and service requests, if yes, kubernetes doesn't care".
Failover by letting Kubernetes do the dirty work is, of course, an option. If the master or a slave dies - it'll come back up. Except... it's also at odds with anything that handles its own clustering model (thinking Redis Sentinel or Redis Cluster, or InfluxDB when it had clustering support). Sometimes "coming back up" in a system that has an orthogonal view to Kubernetes is challenging.
It also doesn't accommodate for the situation where... what if Kubernetes doesn't bring it back up for some reason? If the pod was killed for resource contention, or something along those lines. And now I have two slaves, no master, nowhere to feed writes?
I dont't complain too loudly about these things, because most often the solution is (correctly) to think differently about it, but there are real-world considerations that don't always fit perfectly with the existing Kubernetes model - but we may still have to use them from time to time.
Think of PetSet as a design pattern for running a cluster on Kubernetes. It should offer a convenient pattern that someone can use to Do-The-Right-Thing.
The hard part of clustering software is configuration change management. If an admin adds a new member to the cluster by hand, they're usually correct (the human "understands" that the change is safe because no partition is currently going on). But an admin can also induce split brain. PetSets on Kube try to make configuration changes predictable by leveraging their own strong consistency (backed by the etcd quorum underneath the Kube masters) to tell each member the correct configuration at a point in time.
The PetSet goal is to allow relative experts in a particular technology to make a set of safe decisions for their particular clustered software that behaves exactly as they would predict, so that they can make a set of recommendations about how to deploy on the platform.
For instance, CoreOS is working to provide a set of recommendations for etcd on pet sets that would be correct for anyone who wants to run a cluster safely, and encode that into tutorials / docs / actual pet definitions. They're also driving requirements. Other folks have been doing that for elastic search, zookeeper, galera, cassandra, etc. PetSets won't be "done" until it's possible to take the real world considerations into account safely when writing one.
I'm following PetSets very closely and I think they're going to help a great deal.
Is it accurate that PetSets were introduced to compensate for the type of thing I've singled out? That certain models just don't quite fit with the "everything is equal and interchangeable" notion? Or does it just feel that way? I don't want to end up missing the point of PetSets or using them for something they're not truly intended for, or leaning on them only to hit a stumbling block.
There is likely to be more needed (on top or around) because we need to also solve problems for highly available "leaders" and active/passive setups. Think of what something like pacemaker does to ensure you can run two Linux systems in an HA fashion, and then map the necessary parts to Kube (fencing, ip failover, cloning, failure detection). Still a lot of hard problems to solve - PetSets are the first building block.
If you're particularly interested in Python webapps, I've written a tutorial on how Kubernetes can help deploy a Django app in a scalable and resilient fashion: https://harishnarayanan.org/writing/kubernetes-django/
Upcoming book: http://shop.oreilly.com/product/0636920043874.do (so up-to-date it's not quite out yet)
Tutorial: https://kubernetesbootcamp.github.io/kubernetes-bootcamp/
How many containers are running on Elastic Beanstalk and ECS?
I'd wager magnitudes more are running containers by reading the docs and clicking around than those that mastering the cutting edge challenges getting running on Mesos, Kube or Swarm.
Another blind spot is Heroku. Every day new businesses spin up containers with a 'git push Heroku master' and don't even know about it.
All these providers, platforms and tools have their place.
I simply don't think the "winning the war" talk is accurate or constructive.
Disclaimer: I worked on the container systems at Heroku and now manage 100s of production ECS clusters at Convox.
Just curious, I was skimming through ECS docs last night and it seems like some of the basic building blocks look very similar to their equivalent constructs in Kube/Nomad.
Container orchestration brings in new challenges like operating a consensus database, managing data volumes and load balancing.
And this is on top of VM management.
Yes, ECS, Kube, Nomad are very similar. All relatively hard to use right now compared to their simpler ancestors.
Here is more in depth info about the challenges of ECS:
It shows 32 blocked requests though (ouch).
Quick check:
* cnn.com: 30-40 * bbc.com: 10-15 * washingtonpost.com: 10-25 * medium.com: 1-2 * teccrunch.com: 25-40
The future of technology is held in the hands of open source projects.