Hacker News Reader

Comment by bontoJR | Hacker News Reader

I went to a couple of events where Docker was a topic in Europe, the general feeling is that the solution looks great, but every single time a speaker/organizer asks to raise a hand if using it in production, nobody does. I mean, it's definitely a technology having a momentum, but I still have to find someone that heavily used it in production.

View on HN

That's because it isn't ready for production yet, and certainly not at scale. You're in for a world of hurt if you try to deploy hundreds of containers at once from a central registry. And there's not yet a way to bind containers to physical interfaces that DHCP themselves at runtime.

Docker solves problems for developers, and soon it might solve problems in production. But for large deployments, that day is not today.

Well, we're currently running 80+ instances in my company using Docker to deploy every app. I'm the responsible for the push to this Docker-only approach, given that we had a bunch of legacy apps that were running in Chef-managed nodes (that I've also implemented), so I don't think that it's a developers-only tool, it's already pretty mature and I've not yet run into requiring to bind a container to some physical interface or whatever, the virtual interface has been great thus far.

How large are your images? How long does it take you to fetch them from the registry?

Binding to a physical interface is required if any of your apps will share port numbers on a host. If you don't have network ACLs that control traffic between VLANs, and you have a discovery service to manage rendezvous, then maybe you can get away without it. But many of us cannot.

Our images range from 600MiB to 2GiB in general. When you say share port numbers what exactly are you referring to? App X listening to port M in eth0 and app Y listening to port M in eth1? We don't really have this use case so for us it has been pretty simple.

I think that your use case is a lot more complex than ours or I'm misunderstanding some part, are you using bare metal servers in your own data center? I'm trying to grasp what's your architecture like so I can see what parallel can I make to ours.

You didn't mention how long deploys take. 2GiB x 80 servers = 160GiB; assuming a 1Gb/s interface on the repository, it'll take a minimum of 22 minutes for all the servers to get their new bits.

Not every server runs the same applications, we mostly do deploys on 5-6 servers at a time, it takes around 3 minutes to deploy our biggest apps, most of them deploy in under 30 seconds.

The nice part about Docker is the images are composed of layers, so you only have to re-download a layer when it changes. In practice, we pull all of our base layers when we bake our AMIs, so spinning up a new instance is pretty quick and each new deploy only has to update the top layer that contains our code.

Honestly, the only issue when it comes to image size is the initial developer pull either when they onboard to a team or when they bork their boot2docker install (which, disturbingly, happens way too often).

Many applications are very large (consider a Rails app with all of its prerequisite Gems). Granted, this isn't Docker's problem, but it presents practical distribution bandwidth constraints that Docker doesn't currently solve well.

600 MiB? That more than a complete OS install. Isn't an image supposed to be one application only? Do you store data in them, or why are they so big?

Not sure about sharing port numbers - you can do something like:

  docker -d -p 80:8000 repo/container

isn't this good enough? Can you give more detailed example?

How does that help? All this does is establish a local PAT; you can still only bind one application to port 80 on on the host. If you have more than one service that needs to listen to port 80 and be reached by clients, you're still out of luck.

This is a pretty easy solve...haproxy + consul + consul-template allows you to spin up a new container, register it with consul and have the consul-template watcher regenerate the haproxy config to route to the new port.

It's a little overkill if you're running it on a single machine, but in a real production environment, it's a pretty lightweight solution.

Easy in theory, but in practice, you have to configure and monitor it all. Adding components and orchestration logic is not cost-free.

Well it's not so hard. If there is good document steps for it. It's really abut getting consul working right.

Did you have to change anything during the implementation? What about habits compared to deploy on VMs/DSs? What's you experience regarding a long running? Did you face any issue?

We had to change how we managed services, I think it was for a better way, we were too much reliant on instances being available all the time, although I tried to always think with an HA/redundancy mind most of our team failed to do that. We're now using Docker with CoreOS (and fleet), so it has not been without headaches (mostly in part from CoreOS and btrfs and fleet scheduling going afoul), but we're much more resilient.

Compared to VMs we adopted a lot more of a "this is disposable" mindset, we persist any data that must not be destroyed in a failure in an external volume and treat the instance and service as something unreliable, pretty much it's always "Game Day", before deploying something we've already tried to cover the basics of HA (what happens if this service suddenly stops? How is the rest of the system behave? How do we make it automatically restart or reprovision?).

As I said, most of our issues regarded CoreOS (especially running with btrfs, getting out of space for metadata was a pain in the ass for months) but right now, after moving to ext4 + overlayfs we've been running without issues for a combined 28,800 hours of instance time (roughly NO issue in half a month in an environment running 30+ deploys a day).

I've not stated that we're perfect, very far from it, but Docker itself has not been a pain in production. Our private registry is currently running in an Ubuntu-based instance for more than 6 months non-stop (and we push/pull a lot of data from it everyday).

Compared to VMs we adopted a lot more of a "this is disposable" mindset

There is nothing particularly magical about Docker that means you can't consider VMs equally as disposable. It's a mindset change about how you build your product, not something inherent to Docker. It's the original concept behind AWS's EC2 - treat your VMs as disposable, and store state elsewhere.

You are correct of course but in my experience with Docker so far, it's much easier to get into this mindset, the more ephemeral your "disposable unit" is insomuch as VMs tend to stick around longer than Docker containers and occasionally you might find yourself saying, "this is ok for now" but your barrier for that is higher if you have to deal with things more often in flux.

We're running it in production as well, very heavily (100+ servers running many more containers). The parent is right. Docker is not really ready for production yet.

We've had to write (and continue to write) a lot of code to work around limitations and bugs. We've replaced the registry with custom code, written a lot of code to "manage" the Docker daemon, had to do some funny business to make logging work, are constantly integrating other components (such as weave) to work around Docker limitations, and my God the bugs...

I can deal with the limitations. It's the fundamentals that are the real problem (bugs, memory leaks, descriptor problems, aufs issues, the fucking registry). Docker, Inc. keep chasing new features over fixing the fundamentals. That's their choice, but it's unfortunate for us because, well, I have to go restart a few Docker daemons now because of that memory leak, yet again.

I expect these issues to be resolved eventually and there isn't a better alternative (the ones that exist have their own issues) so we take the pain and wait patiently for things to get better.

You're higher on the Docker product and team than I am, but with your discussion of new-and-shiny over working-and-bugfree you've pretty much nailed why I'm not happy with Docker and am hoping Rocket (or something out of left field) becomes a thing. While I'm kind of terrified in general about the current drive towards VC-backed companies providing core infrastructure, I think CoreOS's team cares much more about stability and correctness and I think I can at least come to grips with them.

The usability of Docker gets so much worse as soon as you get outside the happy examples, too. Like, the one that most recently bit us: Docker is considered a "1.x product" and I have to hardcode a registry URI into the Dockerfile instead of something sane like a host-level search path--like all package managers have done for forevers now-and the suggested option by a Docker developer (and I'm not slagging him, he tried to help and I really appreciate that, it's not his fault this is stupid) was to muck around with DNS to redirect a single hostname to different places within my environment.

Couple the constant friction of usage with the security questions throughout the system (single giant garbage executable that needs root for absolutely everything instead of principle of least privilege, and--oh!--looks like user namespaces slipped to 1.7, awesome!) and I'm not feeling confident in my use of Docker. I feel like they'd rather market than fix.

- I am reminded of this article, and the dismissive replies by the author (a Docker employee and committer) here on HN: https://blog.jessfraz.com/posts/docker-containers-on-the-des... https://news.ycombinator.com/item?id=9086751

Every couple months I clone Rocket, build it, play around with it for a few days, and end up back with Docker. We wrote Python scripts that export Docker containers to a format that is very similar to Rocket's spec so I'm very interested in using Rocket as a replacement for that part of our infrastructure.

That said, this is a hard problem and it is going to take some time for all the issues to shake themselves out. Rocket also has a long way to go before they are production ready. I'm hopeful they'll provide a nice alternative in the long run, but today their platform is just as sketchy.

When I say the bugs will be fixed, I mean they will be fixed, but not necessarily by Docker, Inc. The larger community will have its way as it always does.

You could be right, of course. I generally don't trust a meaningful community of contributors to form around a modern-era VC-backed company (which is to say, "there are no Red Hats here"), so I'm much less hopeful.

> And there's not yet a way to bind containers to physical interfaces that DHCP themselves at runtime

Sure there is. Systemd will actually do this for you.

Docker recently eschewed LXC, and I'm still not sure I understand exactly why, but even after the move to libcontainer, systemd is capable of handling this.

I thought we were talking about Docker, not systemd. systemd is pretty powerful, but you have to write the glue Docker is providing yourself.

If you mean a scenario where Docker launches systemd, we don't do that. In my view Docker is just a packaging system; each container's pid 1 is the app we're deploying.

> If you mean a scenario where Docker launches systemd, we don't do that. In my view Docker is just a packaging system; each container's pid 1 is the app we're deploying.

No, systemd is actually capable of running Docker containers directly (even without Docker).

I definitely wouldn't recommend running systemd inside a container unless you're using systemd to launch the container (instead of Docker), since Docker doesn't handle init systems very well, whereas systemd has this built-in. Though you can also go with the one-app-process-per-container approach as well with systemd, which also works.

If you have any citations that describe how to make systemd run Docker containers without Docker, I'd like to see them.

Starting with systemd 219, you can download a Docker image from a registry with "machinectl pull-dkr", and run it with "machinectl start":

http://www.freedesktop.org/software/systemd/man/machinectl.h...

There is a lot community effort being put into Docker extensions to enable production use cases like what you have just mentioned.

It is going through the same path that AWS did, people start with test and CI type work, then move towards production. Europe is behind Silicon Valley in terms of moving towards production.

I'm using it for a financial application in production.

Granted I can go without it, but currently it makes lots of sense to use it since I need the ability to nicely handle dependencies.

I'm using Grunt for handling the creation, destruction and running of the containers. Docker Compose didn't exist back then.

Reply on news.ycombinator.com