Docker, Containers, and the Future of Application Delivery
slideshare.net
slideshare.net
As far as I am aware, folk aren't really writing distributable applications that target the VM up. You can get preconfigured stacks, or you can get standalone apps that you install in your environment.
But nobody's said: "Hey, if we control the app design from the OS up, we can make it much more intelligent, robust and at the same time sweep away a lot of unnecessary inner platform nonsense".
In terms of the slides, my approach is to reduce the NxN matrix by eliminating a lot of the choices. Why write your blog engine to support 5 different web servers when you can select and bundle the web server? Repeat for other components.
It gets better. Why write a thin, poorly-featured database abstraction layer when you can take serious advantages of a particular database's features?
You can't do this if you write under old shared-hosting assumptions. You can do this if you target the VM or container as the unit of design and deployment.
Yes, this is one of my bonnet-bees, since at least 2008: http://clubtroppo.com.au/2008/07/10/shared-hosting-is-doomed...
Back when I was preparing to take an honours year, I wrote an circulated several project proposals. One was to explore the argument above with a constructive project -- writing a blog[1].
An example I gave for the inner platform effect was the Wordpress file wp-cron.php.
It gets called on every request made to Worpdress because WP has no other way to arrange for scheduled tasks to be carried out. So you get a performance hit and your scheduling relies on stochastic sampling. Oh, and it stops working very well when (as inevitably happens) you slather Wordpress with caching.
In an OS-up design, you just delegate this to cron.
Or plugins. In a standard current design, these can't be isolated. In an OS-up design, you can can make them standalone programs with separate accounts that can't reach into and interfere with the core code. No more broken sites from a PHP error in a hastily-installed plugin. Similarly, you can control their access into the database (instead of having a shared login that all code running in the application shares).
And so on, and so forth.
[1] I'm happy to forward copies of the proposal. My email is in my profile.
Of course, you could argue that the whole design of Smalltalk with it's VM is already abstracting away the OS...
Indeed, some of the advantages from java on the server side is arguably that it has its own VM, so that you don't target the OS platform, but the java platform.
--------------
Decision One: I put relational data on one side and “static", non-relational data on the other, with a big wall of verification process between them.
This led to Decision Two. Because the data set is small, we can “bake in" the entire content database into a version of our software. Yep, you read that right. We build our software with an embedded instance of Solr and we take the normalized, cleansed, non-relational database of hotel inventory, and jam that in as well, when we package up the application for deployment.
We earn several benefits from this unorthodox choice. First, we eliminate a significant point of failure - a mismatch between code and data. Any version of software is absolutely, positively known to work, even fetched off of disk years later, regardless of what godawful changes have been made to our content database in the meantime. Deployment and configuration management for differing environments becomes trivial.
Second, we achieve horizontal shared-nothing scalabilty in our user-facing layer. That’s kinda huge. Really huge.
-----------------
http://www.colinsteele.org/post/27929539434/60-000-growth-in...
Remember PHP register globals? Rails 2.x? Perl 4? I'd bet a lot of lazy devs would still be using those if they could just wrap it all up into a container and say "run this!" That's what commercial products do. And they're much worse for security as a result.
Fundamentally, I'd say the solution is to automate testing and installation. Make it extremely easy for a dev to test app A against a matrix of language implementations B, C, D, databases E, F, G, and OS platforms H, I, J. Make it easy to make packages that install natively on each platform, with the built-in package management tools. FPM and similar help with this. Nearly every platform will allow you to create your own package repos. Better tools = better code = more flexibility = less ecosystem dependency
Containerization as a logical separation for security (ala chroot/jails before it) makes sense, but doing it so you can shove your whole OS fork in there and then fail to maintain it seems foolhardy and myopic.
If the application can control the whole stack, then the application author can fix it.
Automating test and install just puts you back where you started: with a gigantic test matrix that will impose non-trivial drag on the whole application's development.
And it's not necessary. It's just ... not. necessary.
Everyone who is running OS X, BSD, Illumos, Cygwin, and basically every non-Linux OS would take issue with your necessity argument.
I also note that you didn't mention the problem of security issues in bundled dependencies, which is the far more serious issue.
I am not talking about an app that is distributed to be installed "on" OS X, BSD, Illumos etc.
I am talking about an app that is packaged to run "on" Xen, VMWare, or maybe docker (LXC) for some cases. Or zones for others. Or jails. Whatever.
The point is that you, the application designer, ask yourself, "what happens if I have total architectural discretion over everything from the virtual hardware up?"
But, rather than the panacea you envision, what I think would actually happens is you end up with a lot of people doing substandard OS release engineering jobs, neglecting security patches, etc.
Or...
Cargo culting around a small number of "thin OS distributions", which is substantially the same as what we have today.
Heck, "total architectural discretion over everything from the (virtual) hardware up" is pretty much the definition of an OS distribution. Am I missing the point here? Is there something about this other than the word "virtual" slapped on there that's unique from what we have now?
Yes: virtual machines and VPS hosting make it possible to bypass shared hosting. That means you needn't write apps which have to aim for lowest common denominator.
Edit: I agree that the approach I'm advocating introduces new problems. But obviously I think that it's still better than the status quo, which is largely set by path dependency.
E.g. many apps can throw away 90%+ of userland. So while they need to pay attention to security patches, the attack surface might already be substantially reduced.
And LXC can, if your app can handle it, execute single applications. There doesn't need to be a userland there at all other than your app.
Now, it brings its own challenges. But so does trusting users to set up their environments in anything remotely like a sane way.
You are right, but the other point is that it becomes the application authors responsibility to fix it.
If you're bundling apache httpd with your app, and there's a security flaw and a new version released, it becomes your responsibility to release a new version of your app with the new version of httpd.
If there are 1000 apps doing this, that's 1000 apps that need to release a new version. Instead of the current common situtation, where you just count on the OS-specific package manager to release a new version.
Dozens of copies of httpd floating around packaged in application-delivered VMs is dozens of different upgrades the owner needs to make, after dozens of different app authors provide new versions. (And what if one app author doesn't? Because they are slow or too busy or no longer around? And how does the owner keep track of which of these app-delivered VM's even needs an upgrade?)
In any case, there's no difference in kind between relying on an upstream app developer and an upstream distribution. You still need to trigger the updates.
And you might have noticed that stuff is left alone to bitrot anyhow.
If not you can install the updated version of the affected package within the container.
The value of not having to support 5 different web servers and 5 data bases etc... is far far too great to ignore.
If anything it will likely reduce a lot of issues and bugs.
Since you are dealing with only one stack you can devote your time to make sure it is rock solid.
Yes, but now it's their responsibility to stay on top of updates for the entire stack, and push out updated containers whenever any part of it changes. Whereas in the traditional model staying on top of updates to anything other than the application itself is the responsibility of the user or his/her sysadmin.
Not saying containerization of apps isn't a promising concept, just that it does require the app developer to take on a lot of additional responsibility.
It's getting to the point where very soon we will have complete clouds on demand - we pretty much already have them, but soon that will be a trivially selected level of granularity.
We can deploy OpenStack clusters extraordinarily easy now with Fuel. Having fully deployed app clouds is pretty much already here, if not just around the corner.
We are. You might want to check out Mirage [1] microkernels, which is an OS that targets the Xen Hypervisor. You write your application code and select the appropriate libs, then compile the whole thing into an 'appliance'. We have big plans for the kinds of system we want build using this and if you'd like to keep up with it, pls drop me an email. The devs are presenting a Developer Preview at OSCON so there'll be more activity soon.
I had in mind to use the facilities of an existing OS and tools rather than reinvention. My other major inspiration was OK Webserver (and through it, "Staged, Event-Driven Architecture").
I host blogs and one of my pain points is that slow plugins hold up rendering.
What if, for instance, rendering is a graph of pipelines, and there's known logic for failing to meet a rendering deadline? So if you have the blog page and the %#%^^ "Popular Posts" plugin is running slow again, it doesn't slow down the whole site. That <div> merely shows old content, or is excluded.
You can then use standard operating system facilities to ensure that, for example, the "posts" and "page" modules get top priority. Then "comments". Somewhere way down the list might be "complicated plugin that talks to five remote servers which crash half the time".
You can do some or all of this within a programming environment. For the common case, only some. But why not use operating system facilities? They're already there, they're battle-hardened, they enjoy universal coverage of the system and are closer to the metal, real or virtual.
Mirage is I think more of a programming-lanugage environment. Some of the facilities I'm point out exist. Some don't. I don't feel like writing all the missing bits from scratch when they're already available off-the-shelf.
You do not of course end up with a standardised interface from this, as it is dependent on the Linux environment you are running in (although a VM can standardise this). So I am also experimenting with userspace OS components, like the NetBSD rump kernel [2] in the same framework.
Also helping here is the shift (gradually) from applications (like web servers) to libraries that you can link in to your application, or full scripting inside (like openresty for Nginx), which addresses your bundling issue. If you are building single function applications that are then structured into larger distributed applications this is much simpler.
[1] https://github.com/justincormack/ljsyscall [2] http://www.netbsd.org/docs/rump/
Most developers are not full-stack aware - nor should they be. Having a pack of generalists drive a product is sure to reduce momentum.
Better that the contract is "here's a platform you can deploy to" than "here's some CPU time, break it and you get to keep both pieces"
All this does is say "while you figure that out, put it in a script". First, it means we can test the dependencies easily by re-running the script to see that it actually accurately reflects what needs to be done.
Secondly, when you're done, you have a reproducible deployment environment that massively simplifies ops and dev: Ops can decide on upgrades, re-run the scripts, have QA run their tests and know the upgrade won't break stuff in production. Dev can make code changes and be confident that what they hand off will actually work in the production environment because they've test deployed it on VMs built from identical templates.
As long as your team can figure out how to deploy the software they write, they can do this. If they can't, you have bigger problems.
Here's a rather contrived example but it illustrates the idea i'm getting at: "Will the inventory agent software be able to login to audit this container environment when they're done building their release?"
Say you have a web app and a reporting app that use the same database (and probably a communications framework - zmq server rabbitmq etc in the middle there as well). How does docker deal with the following:
* The data? I can see postgres, redis or whatever being packaged up into containers, but what about the data that they use? Will there be attachable storage? Will you share some exposed resource on the host? Will the data be another container on top of the database app container?
* Routing. How do you tell your reporting/web app containers "this is where your message bus and database live?"
* Coordination. I'm used to using something like supervisord to control my processes - what's the equivalent in docker land? Replace the scripts in supervisord with the equivalent dockerized apps? A docker file for the host specifying what to run? How do you know if your app that you've run inside docker has crashed?
* Or do you just package the whole lot above up into one container?
edit actually that was more than one :-)
The question doesn't really make sense: The equivalent is supervisord running inside the lxc container.
> * Routing. How do you tell your reporting/web app containers "this is where your message bus and database live?"
How do you tell them in a cluster? This is a problem anyone who's ever needed to scale a system beyond a single server has already had to deal with, whether or not the application is package up a container, and there's a plethora of solutions, ranging from hardcoding it in config files, to LDAP or DNS based approaches, to things like Zookeeper or Doozer, to keeping all the config data in a database server (and hardcoding the dsn for that), to rsyncing files with all the config data to all the servers, and lots more.
Once you are past the procurement and provisioning steps you really need a way to describe how to wire everything together. I definitely haven't solved it yet but I sure hope to! :)
(EDIT: It actually does look vastly improved over last time; not least the documentation)
Specifically, I run a private cloud at work across a few dozen servers and a bit over a hundred VMs currently, and we very much need control over which physical servers we deploy to because the age and capabilities varies a lot - ranging from ancient single CPU servers to 32 core monstrosities stuffed full of SSDs. They're also in two data centres.
When I last looked at juju it seemed to lack a simple way to specify machines or data centres. I just looked at the site again and it has a "set-constraint" command now that seems to do that.
The second issue is/was deployment. OpenStack or EC2 used to be the only options for deploying other than locally. Local deployment was possible via LXC. EC2 is ludicrously expensive, and OpenStack is ridiculous overkill to us compared to our current system (which is OpenVz - our stack predates LXC - managed via a few hundred lines of Ruby script) .
I don't know if that has changed (will look, though, when I get time away from an annoying IP renumbering exercise...), but we'd need either a lighter option ("bare" LXC or something like Docker on each host would both be ok) or an easy way to hook in our own provisioning script.
(EDIT: I see they've added support for deployment via MAAS at least, which is a great)
So the solution is to "batch up" a load of apps into one container and run with supervisor or something as per my last bullet? I had pretty much envisaged a one docker container per application type of model...
If you build full containers, with a single application, you probably still want supervisord inside each container. EDIT: This is because the container will remain up as long as whatever is specified as the "init" of the container stays up, so in this case your app can die without bringing down the container, and something needs to be able to detect it.
> The data? I can see postgres, redis or whatever being packaged up into containers, but what about the data that they use? Will there be attachable storage? Will you share some exposed resource on the host? Will the data be another container on top of the database app container?
Docker supports persistent data volumes, which can be shared between containers, between the underlying host and the container, or both.
Here's a basic example: http://docs.docker.io/en/latest/examples/couchdb_data_volume...
> Routing. How do you tell your reporting/web app containers "this is where your message bus and database live?"
For now, docker leaves that to companion tools - which typically pass the information via command-line argument, environment variable, or some sort of network service (dns or some other custom protocol).
In the near future Docker will include a more standard way for containers to discover each other. Just like the rest of Docker, we will try hard to make it simple and easy to customize.
> Coordination. I'm used to using something like supervisord to control my processes - what's the equivalent in docker land? Replace the scripts in supervisord with the equivalent dockerized apps? A docker file for the host specifying what to run? How do you know if your app that you've run inside docker has crashed?
Docker itself monitors the process, so you could use it as a process monitor - but it's probably better to couple it with a more standard process supervisor, like supervisord, runit, upstart, systemd etc.
There are 2 ways to combine docker with a supervisor: docker on the inside, or docker on the outside. Both are used in the wild, but I think "docker in the inside" makes more sense. That is, have your init process monitor docker at boot, and make sure it's restarted if necessary.
> Or do you just package the whole lot above up into one container?
You can also do that (and some people do), which leads to the "docker on the outside" scenario, with supervisord or runit inside managing X processes. I would only recommend this as a stopgap, if you are unable or unwilling to break down your stack into multiple containers.
In the long run multiple containers is the way to go. It makes your stack easier to scale, maintain and upgrade.
Standard workloads don't require containers. You can have standard workloads on your physical or virtualised hardware. The choice between these is going to depend on a bunch of factors.
I don't see why they need to be bound together, as appears to be the case with Docker? Plus I don't see how that provides any special leverage (as opposed to having the choice).
1. For developer: - your application works exactly the same way on your development, test and production environment because of using exactly the same os/libs/configuration - very fast snapshot/restore simplifies automatic tests and makes them practical
2. For system admin: - configure your system once in order to make docker work, run any docker image (program) with one, the same command - run many applications which do not influence each other in a quite safe way - download and run application without worrying about its dependencies - very low footprint in terms of disk/memory/cpu usage in comparison with standard virtualization
"More consistent" dev/test/prod is great, but I don't need nor want identical. My needs in dev are different from production - e.g. Everything from basic settings, URLs, networking, reloading behaviour to performance tuning. My needs for my dev DB are quite different from those in production.
2. Agree, but this isn't unique to containers (which is probably what I was aiming at).
Low footprint is somewhat irrelevant to "configure once". You can manage this on physical hardware if needs be.
Low footprint is great when you're resource constrained, somewhat irrelevant at scale. Plus a VM gives you other features/tradeoffs in return for that cost. Different circumstances will tend to favour one over the other.
I find Docker useful. Containers and Standardised Workloads are great. I see the utility in both, my argument is I see them as orthogonal. Particularly referring to the "alternatives" at the end of the deck, which seemed unnecessary distinctions for me.
But you'll need a lot of things to be configured identically, and you'll need it to be reproducible (say for when a server fails). That's the point - nobody forces you to have just one identical script.
The containers vs. vm's distinction otherwise makes a difference for one reason only: Because containers allow higher density, it is acceptable to employ them in situations where the cost of vm's would be too high.
This is how it hangs together with standardised workloads: They're lightweight enough that standardised workloads becomes viable for many more use cases.
(In fact, on Linux, LXC is built on top of cgroups/namespaces which are far more flexible than that: Instead of building full containers, you can run individual applications in their own containers, or you can have servers with built in cgroups support that could e.g. do stuff like take an incoming connection, authenticate the user, look that user up and see "ok, this users connection is allowed to use 5% CPU and 10% of disk IO and 100MB memory", fork() and assign the forked process to cgroups accordingly, while at the same time isolating him so the process in question can only see a certain directory, can't see other processes etc.. Heck, you could even give the process its own network, and firewall it so it can only connect to certain specific network resources. )
But physical servers will not have "standard workloads" unless you wipe them everytime you reinstall, or run a system that can guarantee the server is put in identical state. It's not practical for most people to do.
The reason for binding containers or vm's together with "standard workloads" is thus because the key to standard workloads for most people is down to functionality for rapid, reproducible deployment coupled with isolation.
That Docker has so far been tied to LXC is a separate matter, but I frankly don't see that as a bad thing - too many virtualization / cloud solutions are hopelessly over-engineered.
Additionally, a lot of this feels like Docker is taking credit for LXC. Docker is a brilliant abstraction of LXC's obscure native interface, but LXC came before Docker and does most of the heavy lifting.
Also FWIW, on giving credit: https://twitter.com/getdocker/status/357983297114079233
I think some other comments above make a better argument about how the full power of containers has yet to be realized.
But I do get the hype. BSD and Solaris, although they have many interesting benefits, just aren't that popular for a variety of reasons. I won't turn this into a discussion of why they aren't popular, but it's a fact that Linux is much more popular. Most people aren't going to switch even if BSD and Solaris provide more benefits. But now that an important feature has become available for Linux, I can see why people get excited.
It's like Node.js. Servers with evented I/O has exited for years, if not decades. But the fact that Node.js is Javascript made it mainstream and that's why there's hype.
Process control groups exist in Darwin/Mac OS X - I wish you could sandbox packages with private network namespaces and filesystems as well.
This would create a universal container, assuming all major OSes acquire facilities for process control groups, namespaces and chroot.
Disk space is no longer a consideration. The containers can be as big as we want - why not make them run natively everywhere?
Imagine if we'd have a standardized container format on top of Linux, the BSDs including OSX and Windows with Cygwin - hello cross platform client applications with a single installer. Apple and Microsoft could integrate container technology in their AppStores and we could submit the same package everywhere.
The analogies are pointless, they don't convey anything to me.
http://tryrethink.info - over 3,000 happy customers (unique, sandboxed instances) served in 24 hours. With about a day of effort total.
http://nick.stinemat.es - my blog, which is pretty Docker focused because that's what I've been working on since I decided to start improving writing.
This covers basic continuous integration and deployment of an application to pretty non-trivial volume.
What I will say is this. If you're writing a trivial application that you and only you will ever need to work with, in an environment completely controlled by you, and you have a recipe that works - you're right, Docker probably isn't for you.
If you, like me, work with a huge product suite with many buildtime and runtime dependencies (services and applications), with many different runtime configurations, where even automated installation can take up to 15-20 minutes because of the sheer amount of work that's going on, there's a massive massive amount of efficiency to be gained in the dev/test/release/packaging process, let alone the massive amount of efficiency to be gained by the ops team in working in foreign environments.
There are certainly lots of other use cases (PaaS/SaaS are easy,) and those are valuable business building tools, but less interesting to me personally.
Imagine you wanted to run Wordpress on your DO server. Instead of configuring a LAMP stack and setting up Wordpress on it, you could download and run a Wordpress docker container that came bundled with its own LAMP stack. It would be as simple for you as calling "docker run wordpress".Most importantly, its configuration would be completely isolated from that of the host machine.
I really want to get excited about docker, but I guess I just don't understand it. Any links to more specific use cases?
Stuff like apt-get works great until it doesn't. You've run into this no doubt. Jails/LXC abstract dependencies away in an attempt to avoid this class of problems.
In other words: this helps to avoid mutating your system state with every command you run from the shell. :)
$ docker download-install-and-run-my-customized-build-of-postgres
$ docker start-up-a-wordpress-instance-from-some-online-build-template
$ # edit some config files, add some plugins
$ docker pack-that-wordpress-instance-into-a-container
$ ansible all -a docker download-and-run-your-wordpress-instance-on-all-your-app-nodes
Anyone else experience this?
docker run gtihub.com/some/project
That command will clone, build and execute a container based on the contents of a repository by simply dropping a Dockerfile in the root of the project. That's a fundamentally different level of usage beyond any existing container technology.