Docker will change everything
sathishmanohar.com
sathishmanohar.com
Bullshit. How do I run it on RHEL 5, because DoD systems are chock full of those? What about Windows? I have to install Virtualbox, and Vagrant and then run Docker inside the virtualized linux environment and then run docker containers in that.
So "virtually anywhere" means on a handful of systems.
Sometimes technology's biggest "fans" are its biggest enemies without even realizing.
One of the things Docker needs to get better at expressing is that A Docker container != Linux Containers. It may currently Build/Provision/Execute inside of a Linux Container (provided by LXC) but that is an (albeit important) implementation detail.
The question seems to be this: does this container model represent a step forward? I think it does. (And is it hyped a bit too much? Probably :-)
So do you agree that it runs "virtually anywhere" is bullshit or not. I say it is. Is Windows antiquated too? What about Mac OS X?
It is harsh vis-a-vis the claim it makes. If it makes revolutionary claims, it better hold up to harsh criticism.
> Docker legitimately represents a new idea, and some claim,
I agree with that. So "Investigating container models for application deployment" title?
It doesn't has the same isolation guarantees (yet) but it lets you use the same workflow.
Windows is different, but I can't imagine a scenario where you have to run your app on Windows. I.e. if it's your local machine, you can spin up a VM; if it's in your datacenter, you can use Linux. If your local policies are "we only run Windows on our servers, screw Linux!" chances are that you wouldn't run Docker anyway, even if it ran natively on Windows :-)
I don't agree with that it would run "virtually anywhere" but it should at least run on every platform that has binary compatibility with the platform that a container was built for? Getting docker to work on Windows might take a while but Mac could work.
I actually do think Docker or at least the container model will change a lot, i'm considering staying with Linux as my development platform instead of Mac and that's a thought i'd have laughed at two years ago and the sole reason is Docker.
Solaris has had—for years—container based virtualization with local virtual network and de-duplicating file system snapshots you can tar up and send across machines.
It's new in that they made a nice abstraction on top of the Linux clone of Solaris containers then marketed the crap out of it so everybody became aware of non-VM compartmentalization and fine grained resource allocation.
In short: still awesome, but not "they invented it" new.
But I was wrong. Start playing with it for a while, and you will realize that Docker is not just about jails on Linux. Docker is about making portable, self-contained apps easy. It's about making sharing those containers easy. It's about making using those containers easy.
The core technology consists of a jail, yes. But once you've worked with the Docker registry and the Dockerfile (and its wonderful caching mechanism), and experienced just how easy it is to get something up an running, you will see that Docker > jails. Docker is much more than that.
Specifically, docker lets you build, define, and script new instances, and then to install and launch those instances with a single command, snapshot them, re-publish them, and so on. That's the real secret sauce behind docker.
I think lxc+cgroups and Docker are a much less wasteful way to provision lots of similar systems with particular types of workloads, but this article did seem a little hyperbolic.
So deploying an application as a container is a big improvement over shipping it as source or package, because instead of having to support any combination of library versions, distros and kernels, it only needs to worry about the kernel which is the easiest part.
RHEL has always appealed to IT departments and server staff who value stability over functionality, much like corporate IT at the hospital where I worked finally approved Windows XP SP1 for test deploys, well after SP3 had been released. They don't want to risk the headache, so they stick with what works rather than providing functionality (or compatibility) that others could need.
There's definitely value in that, don't get me wrong, but my point is that the types of environments where RHEL5 are the norm wouldn't be rolling out Docker instances even if they could, and I don't feel as if they should be counted against the impact of progress.
I am not arguing that I need Docker to run on RHEL personally or I need it to run in Windows. The point was about making unsubstantiated claims about a product and in the meantime while seemingly trying to help "market" the product now putting the creators on the defense having to backpedal. "Well no, contrary to what some of your may have read from our zealous followers, we don't don't run virtually anywhere. Here is a list of Linux distros we run out of the box on. Here is a list where you'll need to installed a custom kernel."
If I recall it was an acquisition for them 4-5 years ago.
It's actually promoted like that on their website:
It doesn't make it less bullshitty. It changes that fact that it wasn't only a zealous fan driving up some excitement, they are doing it themselves. Well,I don't like it. I don't like when tech companies selling to developers mislead them or make over-inflated statements. Let 2am infomercials do that. MongoDB was (is?) in the same category, and I don't trust them and don't like their product.
Can you imagine Linus writing the new kernel version announcement like that -- "This kernel release will run anywhere. It will radically change how you look at the world. It might bring about peace too". What over-inflated claims do is break trust. If they are marketing for non-technical people, yes general statements work. Marketing to other technical people using claims like that is patronizing and disappointing.
* We can make the "docker in a vm" option much better. In another thread I mentioned boot2docker (http://github.com/steeve/boot2docker) which is the smallest possible VM to run docker - currently 25MB. Of course there is a performance tradeoff but for, say, a developer's mac or a server which is underutilized anyway, it may be exactly what you need.
* We are adding swappable execution drivers to docker, so that you can replace lxc (the current default) with openvz, or gracefully degrade to a simple chroot. Also a tradeoff - obviously you get very little guarantees of isolation, but for production servers which only run trusted payloads and don't need to extra resource isolation within each server, it is a great option.
In both cases, you still get the benefit of a unique (as in byte-for-byte unique) payload which can be transferred across machines and executed with strong guarantees of repeatability and consistency. That is often a big improvement from shipping provisioning scripts which will dynamically pull dependencies each time, giving you "good enough" repeatability at best. Maybe you forgot to pin a dependency and the mirror was updated between 2 deploys? Maybe the python build on Fedora (staging machine) is not the same as on RHEL (production). Docker solves that. Yes, there are caveats and it's not a silver bullet. And yes, god knows it is over-hyped. But the claims are not bullshit :)
It would be better if you put a footer or a developer's view of the technology, otherwise overarching claims about its features could back-fire. MongoDB over-marketed themselves vis-a-vis the features it actually provided.
I think there is no shame in saying we use LXC container technology. We can run on these systems patched, on these with a custom kernel, on these inside a VM. "Virtually anywhere is a bit of a stretch" I think.
* We are adding swappable execution drivers to docker, so that you can replace lxc (the current default) with openvz, or gracefully degrade to a simple chroot.
That is a cool feature, I like it. Libvirt does this for regular VMs.
> In both cases, you still get the benefit of a unique (as in byte-for-byte unique) payload which can be transferred across machines and executed with strong guarantees of repeatability and consistency.
Good stuff. That sounds like a good plan.
What about a Docker cluster (or maybe you already have that and I am showing my ignorance here). Run docker on a variant of host platforms and docker containers could contain tags that identify what kernel/distro/arch is needed and it would be automatically started/migrated to that host. But it would done rather transparently.
Low resource VMs will be something hosting providers will jump on as that is direct money in the pocket (can put more lightweight VMs on a physical host). But instead of having a generic set of uniform host, they'd just provide the right mix of dist/kernel/arch flavors, underneath. "Oh look people love their Ubuntu 13.10 servers, quickly add more Ubuntu hosts" stuff like that.
> Docker solves that.
Docker is good stuff. I like. My criticism was not of Docker's product but its marketing.
I didn't write this particular article, but I do write our release announcements. Here's the last I wrote: http://blog.docker.io/2013/11/docker-0-7-docker-now-runs-on-... . If you see any bullshit in there, please let me know.
My problem is this: On the one hand, Docker's success so far is engineer-driven. If you visit our IRC channel or follow our github repo, you'll see what I mean. This week we merged 71 pull requests from 44 people. Real engineering teams are using Docker and giving presentations about why it makes their jobs easier. Our marketing consists of helping them meetup in the smae place and show each other their latest hacks.
On the other hand, there is a lot of demand to democratize the story behind docker and containers in general - to explain in plain language why they are important and can change the way software is built and distributed. Typically these demands come from engineers who need to convince their management to allow for broader use of Docker. As a result we produced the now famous shipping container analogy, produced a high-level slide deck, emphasized the high-level advantages of Docker, etc. This has helped a lot of non-experts to realize the importance of what we're doing and how it will affect them.
As a result you get short, appealing sentences like "with Docker you can build an application once, and run it anywhere". Of course, there are caveats - there always are - this or that machine cannot run it, certain versions of Linux don't support it etc. etc. Likewise you could argue that shipping containers can't actually be used for all freight - you can't ship live animals in them! Oil tankers don't use shipping containers! Those arguments are correct, but don't change the fact that shipping containers have changed the World economy in a big way - and so will containers.
My dilemma is that, as an engineer myself, I sympathize with you, and frankly I want you, my peer, to like my work. But every project needs marketing - even Linux, which certainly wouldn't be what it is today if many people didn't spend time and money explaining how it can change the World.
Long story short: I think we engineers live in the perpetual hypocrisy of denouncing marketing without realizing just how much we depend on it.
Working on it :)
Certain elements are already in place - for example docker already has a remote api, and we introduced a "links" feature to connect containers to each other. We are still fleshing this out, but in the coming weeks you should see a robust end-to-end solution for standing up an entire cluster of docker hosts and interact with them as a cohesive unit.
What's the status on accessing other system resources from containers (e.g. GPUs)? There was some work being done to enable this, but I'm not sure if the eventual solution was "run the container with less security so that it can access the cards", or if there was a more graceful approach
Once the image is created you can send it out to any machine that has docker installed and start it up.
The underlying technology is LXC. It is very cool stuff.
But the top article is full of bullshit. It doesn't run "virtually anywhere", it runs on a few systems that support LXC or you have to build a VM that does support LXC, so you can host your Docker VM, inside that VM.
How does that "Yo Dog" meme go...?
So Linux systems, that started shipping just this year?
Well but even that is not true because it was backport-ed to RHEL 6/CentOS 6 which is very cool.
Not all people live dangerously and install the very latest release on their server.
Ubuntu LTS (which is a common server platform) also doesn't natively support LXC well. It has an older kernel. So instructions for installing docker is to install a new kernel. A new custom compiled kernel on a production system. Hmm, is that still Ubuntu 12.04 LTS, some devops will argue it isn't, some will say it is fine.
I would rather see that too. So we agree there. If you read my comment, it was about making ridiculous statements about compatibility, not that I want or need more compatibility.
On a more general level it is about being dishonest and how ardent fans are enemies in disguise. They make ridiculous statements about the product that then creator of the product end up having to manage the PR.
Come on, it was obviously their attempt at explaining it using familiar terminology. It's definitely closer to a VM than it is to a configuration management/orchestration tool.
Does your Dockerfile do anything besides call Salt? If so, what's a good rule of thumb for what goes into the Dockerfile as opposed to Salt or any other config management tool?
Here's a simplified version of the dockerfile template we use: https://gist.github.com/rgarcia/7917826. It basically outsources all of the work to a salt state.
We have a large set of well-tested ansible scripts that we use for deployment, so we thought we'd use the "run ansible within the Dockerfile" approach. However, when we started doing that, we found that the configuration of each docker is actually fairly simple so we converted to (almost) pure Dockerfile. The ansible complexity was configuring and setting up a bunch of images and their configurations on a bunch of servers.
IOW, we use ansible to schlep around & configure the environment variables for the containers rather than creating them.
So it is really a different approach to the problem space that Puppet and Chef have attacked. Docker only addresses the building blocks aspect of the problem, and leverages Linux kernel features that most sysadmins have not yet tried to incorporate into their systems. You can either choose to not use a server orchestration framework with Docker and stick with ssh and bash scripts. Or you can choose a lighter framework that is easier to learn and operate like SaltStack or Ansible. Those of use who have run Solaris systems in the past, know the advantages of containerization in partitioning system resources so we tend to be attracted by Docker.
- Why use ENTRYPOINT rather than init.d scripts?
- When I'm building the Dockerfile, how do I do something like `vi /etc/apache2/sites-available/my-app`? In other words, how do I drop a whole config file onto the image, the way I'd use Chef files/templates?
- Is it possible to parameterize a Dockerfile, so people can use my image but adjust the nginx timeout time?
- Do Dockerfile commands need to be idempotent? For example will this cause problems?: `RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list`
- If I'm using docker to host Rails-on-unicorn, do I still use Capistrano to push a new deploy, or do I spin down the whole docker container then launch another with my latest code? If the latter, are there any existing tools for that?
Previously, to control an environment for an app or set of apps, you either dictated exactly the environment os version and libraries etc, or you tried to run some kind of wrapper like fabric and virtualenv or maven or chef or whatever. And it keeps breaking for real, and developers keep screwing up because they aren't running the exact deployed system.
Docker turns that around. It let's you specify the environment exactly, yet also treat that container as an app. It sorts out the versioning (git), and it makes it trivial for all devs to run the stack identically to production.
As I've just had to get oracle SDK working troubleshooting on my macbook, I can't say how appealing the docker way sounds.
With Docker, the metaphor gets extended to a dwelling with more communal space and shared resources (like a hotel or some type of commune). You have one large kitchen, one large laundry room, and a little space for any other amenities, but each room is its own unique dwelling that can utilize any and all of the external utilities.
EDIT: Really, the metaphor is better if you use chroot instead of VMs
If by "full-container" you mean booting a full /sbin/init in a container, Docker supports that. Just 'docker run /sbin/init' :)
> I'm not yet comfortable upgrading to a new version of MySQL on a database using Docker
Check out volume sharing `docker run -volumes-from`, it allows you to share the same data volumes between several containers. You can use that primitive to upgrade a database, as well as to run short-lived jobs which affect the dataset: migrations, backups etc.
I didn't know about the volume sharing. It sounds cool, but I'll need to think more about how such a workflow might work in production. My initial thought is that such a database upgrade would be considerably more difficult to coordinate than say a yum/apt package upgrade. This is unfortunate but it's not entirely a deal breaker (with proper DevOps, anyway).
The full container concept is only partially implemented it seems, per the issues linked below. Parity with LXC for a more full VM-like experience would definitely motivate me to adopt Docker quicker in more places (e.g. stateful tiers).
Any chance these issues could be given high priority?
https://github.com/dotcloud/docker/issues/1960 https://github.com/dotcloud/docker/issues/2276
Thanks, Joshua
Anywhere that is Linux.
It's a 25MB vm which boots a working docker installation in 5 seconds. I use it on my mac with virtualbox, and connect to it from the native docker client.
That way I can run unmodified docker containers almost anywhere :)
It doesn't work on typical cheap VPS servers from http://lowendbox.com/ because they use OpenVZ.
I wonder a few things:
1. Could tighter security be implemented by outsourcing certain functions to Docker--the base computation can only allow certain routines to run, for example, disallowing any trojan activity? (Similar to blocking ports, but at an IO or computing level).
2. How is this tech beneficial to the consumer market? As the consumer market goes the business market (as we saw in Microsoft/Apple/iOS adoption--consumers demanded it then wanted to use it in business situations)
3. Is the tech Docker using propietary, trade secret, or protected by patent? In our current environment tech firms face terrible regulatory uncertainty due to crappy patents. Can a viable competitor to Docker exist?
Continuous Integration / Delivery / Deployment, Embedded systems, Software/Service Distribution, PaaS, SaaS.. A shorter list would be what use cases is Docker not a candidate for?
> Beneficial to consumers
Efficient Ops allows for innovation to reach consumers faster and more often.
> Proprietary?
Nope.
Docker is open source and open design. Want to change the direction of the project? Open a pull request and get 2 core maintainers to agree. Want to change those rules? Open a pull request to change project governance and get 2 core maintainers to agree.
So I have an IRC channel I can contact the developers at all time, and rollout of upgrades is automated? That's good I guess, but hardly groundbreaking.
If I'm Joe C. ComputerUser with Windows XP running on a circa 2007 Thinkpad, what is Docker and why do I need it?
Good to hear it's open source.
Docker isn't Excel or Word or Some 15 year old game. It's server-oriented software/technology and thus, from a usage standpoint, targeting that community.
I mean, isn't this the economic equivalent of "if a tree falls in the woods and no one is around, does it make a sound?" couched in technological lace?
In fact much of what Docker claims to deliver is actually delivered by some capabilities that have recently been incorporated into the Linux kernel under the title LXC (LinuX Containers). It is similar to Solaris Zones and to the non-kernel features of OpenVZ. Docker just makes LXC a bit easier to use, but you could actually do the same thing with just plain LXC. You would have to figure out some configuration format to use and write some bash scripts. And you would not have the benefit of the community including shared Dockerfiles and the work that Phusion has done on solving some LXC problems.
What docker really buys you over individual instances is lower resource overhead devoted to OS operations.
It's useful to be able to subslice your provisioned resources. That said, you can do it with plain LXC, no need for Docker.
It's just a nice way to package components and make them run together in a predictable way and. That's the aspect I miss most from the discussion around Docker.
Yep
> Sepeate servers?
Today, manual. Docker Host clustering is a feature that needs to be worked out.
> Configurability of items inside of containers
The `link` primitive in the 0.7 release is the key to this one. While primitive currently, it's already extremely powerful and will only get better.
Is there already a project out there that does docker-based continuous integration?
* With docker build you can automate the building of a docker container from source. Simply add a Dockerfile to your source to specify exactly how to assemble it, step by step. This can be done with the high-level build or configuration tool of your choice.
* With docker history you can point to any docker container and get a full record of exactly how it was built.
* We are adding image signature to allow verification of the origin of any particular container image.
If I understand correctly, this is only up to the base image. All the docker docs talk about basing images from well known images from the index. I'd very much like to know how "ubuntu" is prepared for example.
The source, in turn, can either 1) specify a base image to start from (keeping in mind the base image can recursively be itself built from source), or 2) start from scratch and contain all the starting bits.
Of course there is always a "turtle at the bottom". That problem exists regardless of Docker - for example when you download binary packages from an apt mirror, that package has been built from source - but on which build environment exactly, with which build dependencies etc? Docker simply makes this question more explicit by forcing you to ask that question every step of the way - what is the base? - instead of sweeping it under the rug, which we do every day when we install binary packages.
What we're doing for the docker standard library is exposing exactly what the turtles at the bottom are: https://github.com/dotcloud/stackbrew/tree/master/library
Why not just run them on the same "instance"? What does instance even refer to?
It's a 23mb ISO that boots in 6s and allows you to get started right away.
In fact, I think you should run multiple processes because running just a single process is wrong. It will not enable many kinds of important system services, such as syslog. Take a look at https://github.com/phusion/baseimage-docker for a full explanation.
This is also the reason why I wrote baseimage-docker: to make it easy for other Docker users to get things right. I highly encourage people to use baseimage-docker.
> And having separate ENTRYPOINT and USER directives suggests that if you want to run several things at once, they must all run as the same user.
It does not. That's not how the Unix process model works.
When you boot the Linux kernel, it also wants you to specify a single entry point. The kernel starts that process as root. Yet your Linux system clearly has multiple processes, running as different users. Why? It's because the entry point is the init system, which spawns other processes. You can do the same thing inside Docker.
It's exactly the same as how your host system works! Nothing has changed!
I look at Docker as a lightweight way of modeling infrastructure and services. I can built a shipload/stack of containers that model/mock/replicate my application architecture or my SOA model or my three tier web application.
At the crudest level I can do this by modeling each host (web/app/db for example) with a container (hence single service) but it has the potential to scale up to large, clustered SOA applications too. Having that local model of my application means I am developing something that is a lot closer to reality than running a bunch of services on a shrunken set/consolidated set of hosts (or worse on my desktop) in a dev/test environment.
Apologies for confusing the situation.
Also, while you're likely to find pre-packaged dockerfiles for each of your components, the odds are lower that you'll find an existing dockerfile that has your exact stack all-in-one. So, while the "one container" approach may save you some cross-container configuration overhead, it will likely cost you extra dockerfile creation time.