Docker is weird
codeandcake.dev
codeandcake.dev
I'm happy to answer any questions you may have.
What’s funny, though, is that since it was “just” a lightning talk, it didn’t even make it into the roundup of notes I sent to my team after PyCon. But a mere year later, Docker was how my team (and seemingly everyone else) ran projects locally.
Any interesting memories from that day in 2013? From your perspective, what happened in the year that followed that caused docker to be adopted so widely? Were there other key moments or turning points?
That day is forever engraved in my memory... Off the top of my head:
I signed up for a lightning talk expecting an audience of 10-20 people, for a confidential demo of our unfinished project that we had so far only showed in 1-1 demos. Imagine my surprise when I learned Pycon does lightning talks on the main stage!
The previous speaker didn't show, so I started early. In the recording you can hear me joke that perhaps I should get his time allocation too - the MC was not amused.
I went over time and had to speedtalk through the end.. The MC once again was not amused and practically shoved me off stage :-D
We had a small sponsor booth at the even, where our small marketing team was prepared to pitch our product (DotCloud) but was thoroughly confused when a horde of people showed up after the talk, to ask about "Docker" - which as far as they knew was the founder's side project.
I drove down to Santa Clara on my newly acquired motorcycle - my very first trip on the freeway. I remember telling myself, "today is the day to take all the risks!"
I remember my first time trying to use a container was about 2015 on a beach cafe's wifi in Bocas del Toro, Panama. You can imagine how it went...
After a brief hiatus though I was back to it and since then it's been a core part of many a local dev stack and even production stack (compose) I've set up. I recall the magic moment for me being when I first realised you could mount src to use all your local native dev tools while completely virtualising everything else, including and especially whatever combo of runtimes and dependencies you need.
It was just a lightbulb moment, like it entirely solves 'works on my machine', setting new people up on a project, it just seems like how software engineering ought to work and once you have experienced it it's difficult to go back.
I'm curious, as its creator are you still using it, or as someone who can clearly look ahead to experimental ways of doing things are you on to the next thing whatever that may be? If so, what is it? :-)
Yes, I still use Docker. Although my way of using it has changed over time, like many others I assume.
My new project, Dagger, although it doesn't directly compete with Docker, does implement its own container runtime and orchestrator (for CI pipelines), and we recently added compatibility with Dockerfiles and Docker Compose files... So increasingly I find myself using Dagger instead of Docker, in the context of my CI pipelines.
On the other hand, I use docker compose on almost every personal project - like a minecraft server for my kids, home automation, AI hacks, etc. These days I just let my AI agent set it up for me, though :)
“You’re overpaying for CI” (https://www.youtube.com/watch?v=tsH3eTo2GLI) from KubeCon 2024 seems especially pertinent these days with AI.
Was one of my favorite talks that year and really made Dagger click for me.
Yeah I've really been enjoying using AI to write docker/compose config and related ad hoc scripts. It was all pretty understandable anyway, but now you can just set up whatever you can think of in literally minutes, even if you just want a particular configuration as a one off. It's a great era for devops!
Even if you don't need to? Even if there's no benefit?
I was handed a vibe coded project recently, a radio station playout system. For some reason, it's spread across 8 different Docker containers. One container has just Icecast in it, for example.
It's a private system that runs a single station on a dedicated server that we have full control over, but because of all the containers, it's harder to do common tasks like check a log or edit a config file.
OTOH it's a godsend for setting up a local dev machine or something like that, where you need to run different vendor provided packages. but I'm tired of all the mindless over-engineering (which unfortunately the LLMs trained on).
I take it vito talked you into adopting Garden? </injoke>
I was there when dotCloud announced Docker in 2013 and it was truly pragmatic. I had previously burned my hands on LXC, Juju Charms and Vagrant to containerize the apps, but it was all a house of cards. I remember helping a friend ship his whole Vagrant Box to the interview panel for a job!
It’s incredible to think about how much boilerplate Docker removed right at it's launch. Thank you.
That's for sure, especially for local dev.
You may have already answered this somewhere, so feel free to tell me what to search for or if you have a link to hand, that works too :)
Docker in its earlier incarnations was based on openvz, which predates lxc. The lxc project was initially a kernel development project: a collection of kernel patches to add container support. The userland tool was just a companion. When it became apparent that these patches would "win" the race to be upstreamed even though openvz was much more mature, we switched our tooling to lxc. The patches matured over time, but the userland tool (what we now know as "lxc") was finicky.
The original open-source release of Docker wrapped lxc for low-level container operations. But it was just too unreliable and we swapped it out for an in-house Go implementation called "libcontainer".
As for "commercializing": Docker was famously bad at commercializing anything, we open sourced so much (maybe too much) and still people complained that we were not open enough.
To me the arbitrarily complex docker networking makes perfect sense. Especially in a compose setup. I want my containers to be able to talk to each other, but I don't want them on my local network or even accessible (sometimes) from the local machine. The various internal networks offer isolation of containers from the rest of the network stack. If I have a rails app in a docker compose configuration with PG, Redis, Elasticsearch, etc, in a development environment, most of the time I don't want the service/accessory containers anywhere near anything other than the app's container. And if I want several compose stacks to work together, I just put them on the same named network. I apologize if I misunderstood your point.
Which can be achieved with normal routed networks with firewall on the “router” just fine, even on IPv4. Each container can have its own IP address, its own hostname and listen on a standard port, all reachable from containers and from the main OS of the workstation.
I think docker is not weird, it is very linear.
Docker has the merit to take a bunch of tools and to create an organized view of it.
I want to point you to Misterio https://github.com/daitangio/misterio which is a very thin layer on top of docker compose.
It simplify my homelab and works like a charm, and it is a waaaaay easier to use than k8s.
Thank you again, and let me know if you plan a trip in Milan :) for a meeting
People are underestimating now big of a tech shift it was back in the day.
I hope the plug is appropriate in this setting - apologies if not.
In retrospect, of course, it was earth-shattering for the company, despite a valiant effort. The entire expertise of deploying long lived Linux servers and carefully shepherding them through change management with Chef became a relic to most developers. Totally changed the way applications were deployed.
Was an exciting time!
For one, did you get rich beyond all comprehension and did you start a band with your wealth?
Second, do you still program ?
I was lucky enough to get in this industry when Docker was new. Sure beats trying to figure out why X happens on machine A and doesn’t on machine B.
I did not start a band, but I did build a nice house, help my family etc.
I still program everyday, working on my second startup, https://dagger.io (please forgive the plug). We have a very active discord if you want to come say hi.
I guess my follow up question is once you’re financially independent, what continues to motivate you?
Is it fun?
Do you feel like you’re creating tools that the industry needs.
I imagine if I ever get rich I’d work on music and small games.
Video games that you buy once and get to play forever, self host, etc. With a full open source commitment within 5 years or so. I’d probably create a game studio in a lower cost of living country though.
Shortly before this (the weekend after you secured YC funding, IIRC), I['m almost certain I] met you in downtown Oakland at Saagar & Tiffany's /r/bayarea meetup [0] – won't go into further details, other than I was one of their co-conspirators =P
My point of writing this is dualfold:
1) Thank you for sharing your design – I literally just installed my first Docker module (last week) and am floored at all the problems Docker might help me solve (I am not employed in tech)
2) My recollections of you, this decade+ later, was of a humble young man (we're the same age) that didn't know what else to do with his newfound techfame (and money) than keep'on keepin'on.
After watching your Docker Debut video, I'm fairly certain it WAS you: we chatted about dirtbikes (and just what the hell you were going to do now that you "had money" – then such a foreign concept to us both)... if this wasn't you it still seems to echo in your mannerisms/tact/approach. May your humbleness and cleverness prevail!
[0] It might have been at Gustafson's NYE party (I was the guy remorselessly conjuringbitches; "Volcano"); we had a mutual friend at MalwareBytes?
Docker completely changed my life. When I started using it in 2014, I really appreciated how well it solved certain problems I had for web development.
It spawned a near endless stream of contract work and video courses for about 10 years, along with being introduced to the Docker captain's program early on. Met so many great people and had a chance to visit a few countries around the world through Docker.
It's one of those pieces of tech that has followed me around for half my career and will likely continue until the end. It's the best community I've ever been a part of.
And yes the community has always been the best part of Docker...
What was your imagined use case when you first came up with it?
Did you think Docker as it is used in the industry right now is where you imagined it would be used?
Any use cases for it that the industry missed but you think would have made things even better?
> If WASM+WASI existed in 2008, we wouldn't have needed to created Docker. That's how important it is. Webassembly on the server is the future of computing. A standardized system interface was the missing link. Let's hope WASI is up to the task!
How are you feeling about WASI today?
I'm curious if when you were originally designing it, what tradeoffs you chose between, for example did you consider a non-daemonized design or was it daemonized from the beginning?
Having had some time now and also seeing how other people have riffed off of the idea and refined it, what would you have done differently? If you have a retrospective at some point already, I'd be interested to read it.
I have some accumulation of personal architectural guidelines that I built up from trial-and-error + reading others' experience and reflecting on how their design could've affected something I made. As a result, I really value those kinds of discussions about tradeoffs.
I know that containerization on Unix-like systems didn't begin with Docker, but I definitely think Docker was the first accessible interface for it.
I think app containerization is something that has so many benefits and it's the obvious correct approach for its use cases once you wrap your head around it. This was already understood at big tech companies pre-Docker (e.g. Borg, Tupperware, user app sandboxing on iOS and Android), but I think it really took something like docker to make it mainstream.
As far as I'm concerned, you can probably consider yourself like the Ken Thompson or Dennis Ritchie of your era in terms of the impacts of your contribution.
Java App Servers with their WAR file deployments across a cluster were the closest equivalent.
Docker was the enabler for alternatives.
So, not only did I not anticipate this kind of success, I did not even have a reference frame for what success might even look like. All I knew was, I had an itch that I had to scratch, and I had nothing better to do :)
I didnt get it at that time haha.
When Docker came out it solved so many issues we were having. Thank you for scratching that itch.
One design principle that was very important to me, was portability and avoiding fragmentation. I wanted every image name to have the same meaning in every engine, so that when writing a Dockerfile, you could have confidence that it would actually behave the same on every docker installation. In this we disagreed with other platforms (like Red Hat) who sought to "embrace and extend" Docker by changing the meaning of short names. We always fought this to avoid fragmentation of the platform. This did not make sense from Red Hat's perspective, since they were the platform - who cared about fragmenting Docker? That led to some negativity which is probably what you're remembering.
At the same time early Docker users wanted the ability to host their own images without being forced to use our central registry. Using Go's system solved both problems at once.
I personally never use the short names and have really come to like your solution. I host my own registry and I can glance at a build or compose file and immediately know if I’m depending on any external resources that I shouldn’t be.
I originally disliked it, but it’s been my favorite namespace solution for a long time now.
None of these things can be transposed today, so I guess I can't answer...
For what it's worth, my current project (dagger) is written in Go.
Actually there are some longer unedited versions I might release.
Another explicit goal was to facilitate contribution. We thought Go syntax was "vanilla" enough to attract a broader set of contributors than a more niche, opinionated language like Erlang, Haskell, Clojure etc. I think that also worked out well - Docker was known for attracting many contributors, many of them making their very first open source contributions. We even ran special meetups to help people become contributors, that was fun.
Prior to the creation of Docker, did you have experience with other containerization technologies like Solaris Zones and FreeBSD Jails?
Did you at some point, either prior to or after making Docker for Linux, consider building something like Docker on top of FreeBSD Jails?
As a user of both Linux and FreeBSD, running Docker containers and Jails on each of them respectively, it would be interesting to see what FreeBSD Jails could be with something like Docker built on top of it while also taking advantage of other features on FreeBSD like fully using ZFS snapshots, ZFS send/recv, ZFS clone, vnet, and so on. Such that spinning up a container would under the hood receive ZFS snapshots from the “Docker registry” equivalent server, and it might clone the snapshot locally before starting up a container instance so that multiple container instances would be using the same locally available ZFS snapshot that was received once from the “Docker registry” equivalent.
(And as a result that, with ansible, continues to be my preferred way to deploy while the rest of the world passed me by…)
However Moby was a poor choice for a skunkworks project, it involved mostly communications and project management, which requires consistency and scale. Overall it was poorly executed because I failed to build enough support internally ("Solomon's at it again, let him do his thing"). Here I am trying to explain it on Twitter after the keynote, using a picture of a hand-drawn diagram, because nobody in marketing or product was assigned to help. https://x.com/solomonstre/status/855918630915133440
I think the goal of Moby was fundamentally correct: we were trying to replicate the Red Hat model, with well-scoped upstream components (linux -> runc, containerd..) integrated into a commercial product (Docker -> RHEL), with a faster-moving community project in between (Fedora -> Moby).
This was a way for us to clarify that Docker was a product, with a community of users, and Moby was an open source project, with a community of contributors. This clarity was very much needed, because different communities were projecting very different expectations on the brand "Docker", and these expectations conflicted with each other.
In a way, Moby aspired to be what CNCF ended up becoming - the "big tent" where the broader community meets to hammer out the general shape of the container stack. It seems obvious now that a single company can't be the sole authority over such a vast scope. But in the first years of Docker, we really did have that level of influence, because we literally invented the market, and brought the community together in meetups and an ever-growing Dockercon. It's only when Google and Red Hat pushed Kubernetes and CNCF, that our influence over the stack started to wane. Maybe it was for the better, I don't know if we would have been up to the task as designers and engineers. But at least there would have been a consistent design, with a Linus Torvalds figure willing and able to say "no" to stupid and redundant features. For a while I played that role - but then I was replaced by SIGs and committees.
In recent years I have shifted over to Podman, primarily because it defaults to rootless without a daemon.
Is that something you considered for docker early on , and if so why did you decide on running as root?
I also feel lucky that people are still using my work, that it endures without me, and that it generates (mostly) positive reactions. There was a time where Docker had a lot of haters, which was hard on me. But with time I've noticed that the negativity gets mostly washed away, and what remains is positive interactions like the ones in this whole thread. So now Docker is this cool experience I share with lots of people around the world, and when we meet we get to talk about this shared interest. I love it.
I've struggled for years wanting something that wasn't K3S or K8S. Something good for small cluster management that had a lower level of entry, that I'd feel better handing off to other teams.
Docker Swarm seemed to fight the puzzle, but falls short based on things that needed to be fixed or finalized, that just never happened.
Any reason that whole pathway got abandoned? Other than just "Google uses Kubes so just everyone can use it"?
I’ve been following dagger and it’s been really interesting to see the evolution of the design, and new set of abstractions. There’s also nix which has seen popularity, with their approach of building layered oci containers.
The sole idea of containerising an application, its utils and dependencies all into one container is a real big deal in early 2010s. The product has been so visionary that it has been re-pivoted as secure sandboxes which is most needed right now. Kudos!
Why doesn't docker respect unix filesystem permissions? I use docker on a shared server, i can delete docker images of other users. This is bonkers.
Is it a defect of a particular docker setup (following standard instructions on ubuntu), or it's an intrinsic feature of docker itself?
Same reason a Postgres root can delete tables of other users.
It could, but it would be a shim on top of Docker to respect them.
Docker is a service daemon running as root, and the images of other users are just entries created by the other users in its registry, not server files that have permissions of each user.
When you're creating a new docker image, you just send a command to that daemon process - you're not passing your user account credentials to create. As far as docker is concerned, it's just one more image it knows, not an image belonging to you specifically.
Why choose to break user boundaries? Why can't docker images be files in the possession of each user?
All of the above applies to Linux, ymmv on other OSes
The folks at infracoders / DevOps meetups would think we were a bit "out there", then what do you know - 2013 rolls around and we quickly started to shift to Docker (too early some might say), but not long after shifting from LXC to Docker it seemed many others finally started to come round to containerisation.
Fast forward to 2026 and I've recently done some work with a client who is only just "getting into containers" (a Windows shop).
Thanks for your early work to bring better containerisation to Linux Solomon. It's funny to think we're 13 years from Dockers release. Time flies.
Went from the good old days of vagrant (xen/kvm) to lxc/lxd to docker, and docker truly clicked all the way from development to production.
But honestly, thanks to Docker hub even more. The CLI is only one part of the puzzle, the worldwide registry is an immense undertaking.
The most successful project that we worked on together was "project Piñata" which later became Docker for Mac. It turns out bundling a Linux VM in a native Mac app, and integrating it into the host filesystem, networking stack etc requires lots of magic..
Now, most of the Unikernel team were very active OCaml developers, and created MirageOS among other ocaml projects. While working on Piñata, they simply reached for the tools they were most familiar with.
When acquiring our company, Solomon took us to a remote farm in France for a week for an offsite, fed us fantastic food and wine, and the first prototype of that Docker for Mac VM was working by the end of the week! It took a few months to hack together the shell. There's a video of Solomon and the pinãta launch party on my blog ;-) https://anil.recoil.org/notes/cacm-docker-cover#getting-invo...
There was a whole other slice of work we did for project Piñata too in the form of datakit (https://github.com/moby/datakit) that was well ahead of its time. I was looking at this archived code again just last week while thinking about agentic git workflows!
What are you up to these days?
Hope things are going well with you at Dagger(?)!
My memory from that era when Docker was getting big was our biggest user completely ignored us. Which I thought was quite odd at the time.
I did speak to Greg several times.
I think over the years there was natural overlap, people who contributed to Docker would also be kernel contributors. Especially when we spun out runc, containerd, which created a more vendor-neutral space for maintainers to focus on the bottom of the userland stack with minimum drama.
To the extent that Red Hat employees overlap with kernel folks, that made it more difficult to collaborate since Red Hat was overtly aggressive towards Docker, and generally made our engineers feel unwelcome on "Red Hat turf". I discovered that they had devolved into a toxic company with a facade of benevolence. This made me sad because Red Hat was an inspiration to me. One of those "don't meet your heroes" situations.
Not sure what you meant by the "ignoring" part.
We had public discussions on the Linux Kernel mailing list and later on the containers list.
It was a collaborative effort from vserver, openvz, IBM, and Linux Networx, and google. I think those were all the original groups, my apologies if I have forgotten someone.
After Docker made a splash I occasionally asked other developers if they had any contacts at Docker. To the best of my knowledge we never received a bug report, or a feature request. No one from Docker ever showed up at any of our meetings.
I know at one point I wanted to let someone know that Docker they were holding the pieces wrong when it came to security.
Eventually I think one of the OpenVZ guys wound up working for Docker for a time. But that was quite late in the timeline.
I was passing through RedHat about the time of the spat you mention. My impression at the time was that Docker never worked well with anyone, and after some frustrations trying to cooperate the RedHat userspace folks decided it was easier to get a real standard without Docker.
At that point I had gotten the unprivileged user namespace support into RedHat's kernel and changed a lot of minds about it's utility.
I don't remember Docker ever picking up unprivileged user namespace support and as a result was a security disaster waiting to happen.
It felt to me like Docker treated the userspace API for containers like most folks treat the POSIX API. Fixed in stone. In fact there was a lot of wiggle room, and opportunity to shape things for anyone who was willing to dive deep into the technical details.
Docker is probably the highest-profile software written in Go. Are you happy with that choice? Would you do it in Rust if it was available at the time, or Zug?
When it comes to the author calling Docker weird, after looking at some of their other postings, something about glass houses comes to mind.
Was it all for the `docker.sock`?
And multi stage builds are just a directed graph short hand for having separate intermediate dockerfiles. There’s nothing special about them either.
If I had to pick one thing that is tricky for someone new to docker to understand, it’s how the filesystem state is cached using the previous state, the env, and the raw command text.
Docker is much more than memorising a handful of Dockerfile directives.
When I was learning Docker I can't tell you how many people would describe it to me by analogy to a standardized shipping container, and start going off about shipping container dimensions and supersized cargo ships and trucks and railcars... and finish by smiling beatifically, like they had just explained the whole thing in a masterstroke. Let's... uh... not do that. :-D
In a problem set, the student might learn a bit about dockerfiles or docker-compose.yml... but only, I think, so they can try building and running a container of their own!
Docker syntax is one of those things that just immediately falls out of my head when I close the editor. I don't care about it at all.
Even better: there are many, many tools that build OCI images without using the Docker syntax. I haven't hand-written a Dockerfile in at least six years.
LLMs have not loved the needle here in my sphere, it was already bouncing off of the zero on the gauge.
edit: this isn't to say the syntax is good or bad (though there are several questionable parts of it) but extremely simple, and concise enough that you can just look up literally any command with great ease. It's surely a large part of its success, for better or for worse.
Funny how little has changed except the names, I guess.
You're a genius well done making containerization actually happen.
Am I on the right track?
Python and Go are both memory-managed languages. That is perhaps the hardest part of programming (outside of getting the code to do what is intended).
Go and Rust are both compiled, but Rust requires the developer to figure out all of the memory management before compilation, which slows down development to a crawl.
Almost no company will rationally choose Rust over Go if they know how the development speed tradeoff looks like.
I would have chosen C++, but given that isn't the new darling any longer as in the 1990's, that begs the question of the new kid in town that everyone is now rewriting everything into.
So they would probably never rewrite from Python into C or C++, because not cool, too hard kind of motivation, otherwise they would not have picked Go.
So that begs the question, if the obession as you call it, was already a thing back then, would Rust have been the lucky candidate?
Once I realized how difficult it is to write complex code that actually satisfies the borrow checker, I was perfectly happy with writing somewhat slower code in Go or Python.
What made Docker unique were there things:
1. Dockerfile concept. Building images layer by layer. Caching layers. Everything in a single file. It made possible to iterate on image quickly. It made possible to implement CI builds efficiently.
2. Docker network registry concept. Putting universal images onto a server and share them with the world.
3. Momentum. A lot of software that was traditionally complicated to deploy, suddenly appeared in docker servers. I remember how fascinated I was, when I could deploy wordpress website with tiny docker-compose. And how contained it was. I could just delete every trace of it from my host. I always felt uneasy installing various things onto my host and docker just solved that uneasiness.
The main draw for me was that it made containers absolutely trivial. Sure, you could spend ages figuring out how to DIY some isolation, but in practice an awful lot of people stopped at having Apache fork its workers with a non-root uid. You want isolation and a repeatable setup? Spin up another VM and have Ansible set it up! Docker provided a batteries-included lightweight way to isolate and manage multiple applications at once.
Dockerfile? I would've been happy with a Bash script, or handing it a tarball. Caching and fast iteration? Compute is basically free. A single file? Well, there's already a whole bunch of app source code files sitting next to it, so what's a few more in a `setup/` directory? The registry? A shared base could've also been achieved with, say, a `deb` file providing the right dependencies, files, and post-install scripts.
I do agree on the momentum, though. Having every piece of software reduced from a multi-page setup guide to a single command and not risk screwing over your entire machine is absolutely magical.
Layering is as much an abstraction paradigm as a performance one. Its not trivial to verify every loose bootstrap script or .deb that would be needed to build a sandbox... But in a Dockerfile you can see its just some official alpine image base, and a few added files and an entrypoint.
You can absolutely build images with a bash script, here's an example (its vibe slop, but it works): https://github.com/andi0b/vibe-todo/blob/main/docker-build.s...
That script is just calling docker to build layers manually via `docker create`/`docker commit`/`docker run`/etc, using a Dockerfile as a rudimentary DSL over it. It's still building images "the way docker does", but in a more error-prone way.
What GP is trying to say with "I would've been happy with a Bash script, or handing it a tarball", is that it ought to support giving it a literal tarball, like you'd get via `make install DESTDIR=./out && tar -czf image.tgz -C ./out .`. Or just tar'ing up the build artifacts of any other existing, competent build systems. There's no need to treat building an image as some "containerized" thing where you have to run inside the bottom layer of the container to build the artifacts.
In fact nowadays with the best practice of split container builds, there's no benefit of saying "the container is guaranteed to have the dependencies because it's the container you built with", because in practice nobody wants to ship their runtime containers with an entire build environment in them. So they do one container to do the building, then pick out the artifacts they need and place them in the runtime container. At that point, what's the point of using containers to build again?
And for most people, that's a bad idea. Most people don't have to know what RPATH, ldd, multiversion/simlink library discovery, etc are. And even if they did, Dockerfiles give you an easy way to do something you couldn't easily do before: multi-stage builds and a highly robust way to say "drop the parts of my transitive hull that were only relevant at build time". Dockerfile semantics kind of descend naturally from needing the behavior of stages, parallelization, cache keying for image layers, semantic layering (FROM), and making simple use-cases relatively at-a-glance/contained.
Edit: or is something stopping you from doing it this way? It seems like people just don't like Dockerfiles, but instead of just doing it differently (which is possible, it's just a tar file and a manifest), they keep complaining.
Basically both described in https://www.usenix.org/legacy/event/lisa11/tech/full_papers/... (in detail) and https://www.usenix.org/conference/usenix-atc-10/apiary-easy-... (in a "sexy" use case to try and motivate it) a few years prior. (and I get annoyed when people say "so you're saying you invented containers?" because they equate docker with containers. And I'm like no, containers existed for a long time going back to logical partition on mainframes in the late 70s/early 80s. All I claimed was a novel way of building and distributing file systems for them that made them much more useful and valuable and in apiary a novel desktop setup that made use of it to improve desktop security)
Even the original docker implementation of using a union fs (docker using aufs) was the exact same as described.
Even if docker came up with it independently (though conferences like LISA and Usenix ATC are where the papers were presented and were where companies like dotcloud/docker would advertise their wares (and in fact docker inc was an exhibitor there, so it wasn't hidden), the differences between the uniqueness of docker and what is described in the paper are minor (in my perhaps somewhat biased opinion of course), and i was clearly first.
But the argument I used for advocating Docker in environments where people were not fans of it was that the Linux container had become the de facto standard unit of deployment. There is so much tooling written under that assumption, and if you want to use any of that tooling, you need to package your deployments as Linux containers, even if you want nothing to do with Docker otherwise. That's why it had so much momentum.
Then a fourth point I think you're missing is that the concepts used by Docker are relatively simple. Before I understood what Linux containers were, I used to think things like, "Why would I care about Docker when I can isolate my processes with Linux users, cgroups, namespaces, etc." What I didn't know at that time is that "Linux users, cgroups, namespaces, etc." is exactly the thing Docker was giving a friendly UI to.
The first two are just kind of the nature of containers on linux, and applies to any container technology.
Podman does allow you to run it as a non-root user (with some limitations), that was in fact one of the biggest motivations for making podman. This wasn't possible at all in the early days of docker, it wasn't until linux introduced user namespaces, where the root user inside the namespace mapped to a non-root user id outside the namespace that non-root conatiners were possible.
And podman, unfortunately IMO, is also written in go.
Podman, by default, uses crun as the container runtime, which is written completely in C. Podman is basically just a frontend for managing running containers.
Not anymore, they switched to runc (but it can use any OCI compliant runtime)
Otherwise it’s basically the same and you can continue using your Dockerfiles and Docker Compose files.
Unless you’re doing something really fancy it’s basically a drop-in replacement.
Unfortunately, there’s an unconditional setsid during podman’s container setup, which as far as I could tell exists so that the container can be detachable.
(This was all so that I could have the same job management with Ctrl-Z, SIGSTOP, SIGTSTP, fg, and bg, regardless of whether the process I’m running starts up a container. The only way I’ve found to have that is by handling chroot/unshare myself.)
:D
A lot of the more important syscalls involved do require root.
Knowing this, you might find rootless Podman's fork-exec method of starting containers to be more sensible. I certainly do. No root, no daemon, no socket.
If you want to learn more about the Linux Kernel namespaces which work together to make a container, this LWN series is really good: https://lwn.net/Articles/531114/
I worked at Docker from 2014-2019. The most important part is actually written in C.
[I wrote a whole comment here about to explain how the hack works because I was too fixated on how technically incorrect "written in Go" is. Then I went back to read the rest of the article only to see that the paragraph immediately after this kind of explains it in its own way! Let that be a lesson to always finish reading the article before posting about it.]
Everything in that list feel natural when you know how things evolves over many years. Maybe it is just me getting older and less curious.
Now it's possible to create containers without elevated privileges (assuming user namespaces are allowed), but that wasn't possible until long after docker had established the client/server model.
rkt (2013) was CoreOS's alternative to Docker and was daemon-less and failed.
Podman came along relatively late (2018) and benefited from a lot of Linux kernel and systemd improvements since Docker was designed.
So while arguably the client/server model is outdated for container management tools, I'm not sure it's weird. This is my workstation for example:
schmichael@fedora:~$ ps -e -o comm= | rg 'd$' | wc -l
67
[1] "Docker implemented in around 100 lines of bash" https://github.com/p8952/bocker
- Today, a lot of functionality still in Docker has been upstreamed into the (Linux) kernel, so containers can be much simpler (and rootless). Docker is still carrying around some of that cruft, which is what makes it slower and heavier than Podman.
- As a devex technology, Docker/containers have been massively successful, but not as infrastructure. Containers never became secure enough to run multitenant workloads without some other form of isolation, so they ended up being redundant with KVM (or gVisor, or a similar second layer) in production a lot of the time.
That second contradiction led to lots of inefficiency. For example, on Google App Engine, for years when you ran a deploy it would create a VM just to run a single container.
K8s requires some sort of tool to have any hope of storing secrets in a way that isn't trivallly findable from the rest of the cluster.
But, perhaps I am being too harsh, its not like there are secrets primitives in linux that are easily deployable to namespaces/cgroups.
A few cool tricks:
`range shell python:3.12` - opens a shell image without pulling the entire image.
`range shell ubuntu --commands` - you are in Ubuntu. Type any command Ubuntu doesn't have and it comes from the image that has it. On the fly.
`range run --network none ubuntu` - network-isolated sandbox
range run ebs://snap-... -- cat /etc/os-release - opens an EBS snapshot with no EC2 instance
Another cool trick:
`range run python:3.12 --mount hf://moonshotai/Kimi-K2-Instruct:/model -- du -sh /model` says 959G instantly :)
Check it out here - https://getrange.sh
Most importantly, Range doesn't need any daemon, any container runtime, any registry or any image conversion. One binary only. On a Mac it boots its own tiny Linux VM per session, and on Linux it uses KVM.
Range always reads only the bytes your program touches. Nothing else. You can prob shell into a 1TB image from a Raspberry Pi :)
It turns out you can do all the packet processing magic (connection tracking, syn handling, etc) in userspace totally fine, and then indeed hand it over this to userspace syscalls.
Being upset with classic C-based slirp and rust smoltcp (not enough features) I've dug up Fuchsia userspace tcp/ip stack - netstack - and wrapped it in slirp a-like envelope.
https://github.com/cloudflare/slirpnetstack
So this beast is basically that - on one hand you pass it packets (potentially from inside container) on the other hand it does connect/bind/etc to host OS.
This approach, in one form or another, is used by tailscale, podman and many other container runtimes.
One of the gnarliest issues I recall was where network namespaces could occasionally not be cleaned up correctly. So "randomly" (but more likely under certain loads) the networking of a reused thread would get jacked and Docker could no longer pull images.
(rule broken by almost every language in common use, unfortunately)
> rule broken by almost every language in common use
I don't think so. Go/Erlang/Java do break it (Java very recently). C/C++/Rust/C#/JavaScript/Ruby/Python/Perl/TCL don't break it. PHP ... sort of breaks it, (IMO they just hide cooperative nonparallel concurrency in a nice API, but I'm not well-informed on php)?
Because most web servers use OS threads. Nginx, Apache, litespeed, they all spin up `core count` threads but they can all work single threaded just fine as well.
Try to make your own VM runtime next, you might be surprised!
Johnathan Blow's talk on the bronze age collapse pushed me to simplify my production stack. I wanted 'just processes', so I stopped using docker there. I'm happy with the result.
There's a fun little easter egg: the combination of "Steve Wozniak" and "Boring" is explicitly forbidden, because Steve Wozniak is not boring :) https://github.com/moby/moby/pull/4902
Containers From Scratch • Liz Rice • GOTO 2018
Run containers on bare metal already! - Bryan Cantrill - Velocity NYC 2015
This is very important for debugging! Not just that process, but the processes running in that Docker can be found in /proc.
Why this is useful is that you can run gdb outside of that docker and point it to a process in the docker, including setting the file system root to the docker one, so that it gdb reads the correct objects from the correct paths, to get the right symbols and addresses.
For me firecracker has mostly taken over most of my hosting needs, as i can actually take a docker image and unpack it into a firecracker image, and still have a slim process (vm), with great performance, though with a bit extra bloat, but now also have a secure workload that I've got better control over.
Docker and docker compose however, is still the core of my development workflow, and has been for many years, in my eyes, docker compose is the best we've got for simple composition of various services.
Well said. At some point i was toying with doing a http server that will containerize itself on top of the directory it's serving and put it on the back burner because the list of features needed seemed never ending.
Other weird facts about OSes: a filesystem is a database, the Linux kernel is a monolithic program, "hidden" dot-files were a Unix bug that got carried forward, deleting a file doesn't remove the file from your machine, virtual memory is a lie, your computer can't run multiple programs at once (it's just running one at a time real fast), in 12 years most systems will crash due to one number getting too big, you technically never need to reboot because you can launch a different version of Linux while already running Linux, you can freeze a program to a file and then later resume running that same program in the same state (but nobody uses it), files aren't actually stored in directories, there's a maximum number of files (and size of file) on most filesystems (which you can use to lock up a machine, despite file quotas existing and nobody using them), you can lock up most systems by forking too many times / using too much RAM / accessing disk too much, there's a default number of open files and/or TCP ports which is artificially low and you can use to again lock up a machine, pipes are just two file descriptors connected to one memory buffer, an IP address is just one long number split up into bytes, etc.
Of course it can.
My bash build scripts looked like Dockerfiles.
Most of the container code in Linux came from those russian guys.
Took me 6 months to find out the difference between stateless and stateful containers, after Jerome Petazzoni published a blog post opposing the usage of an SSH server inside containers.
It's not a vm: It's all namespaces, baby! Unshare one or more of your mnt, pid, net, ipc, uts, user, cgroup, or time namespaces and you have a "container".
This guy is totally correct. We also used Go for everything -- as it was hot stuff at the time -- and also very frustrating to deal with early version. Dependency management nightmares, bugs in the runtime, etc.
He also had to have a C shim that got loaded first, or shelled out to.
The hype around Google tools was pretty fascinating. I felt like that was the primary reason around using such an immature language at the time. There were better systems languages around for that kind of work.
It lets you package apps using a syntax very close to a Dockerfile. It spawns lightweight NetBSD micro-VMs that boot in just 10ms. hardware sandboxing without security trade-offs.
Is there any writeup about those, or why they are there? Is this just to separate the different "networks" in a compose setup, or is there more going on?
>>Containers aren’t real
To me a Docker container is less like a sealed box and more like a 'beam' of processes running through a set of isolation boundaries.
Though it would be better if it compared Dockers ways to say Podman, which does things a little different I think.
For others, like me:
Footgun:
A footgun is a programming or system design term for a feature, tool, or API that makes it very easy for a user or developer to shoot themselves in the foot—meaning it causes self-inflicted damage or severe errors while working exactly as intended.
Side note almost nothing about Docker is weird. This article chooses rather boring design decisions and picks on them.
"It's an idempotent language for OS level dependencies so you can be sure exactly what is running on your server"
Gimme gimme gimme!
Performance is basically irrelevant because you're not really interacting with the running application. You're running as root, so security-critical. You're talking to the internet for things like fetching images. You need rather complicated business logic to deal with all the possible options, protocols, and file formats.
Sure, you could do it in C, but dealing with dependencies is a major PITA and you're permanently one tiny mistake away from getting pwned, while not needing any of C's strengths. Go is safe, easy to write, and ships as a single batteries-included binary: what's not to like?
This is not a slight on Docker itself. It’s a slight on the conditions that induced Docker to exist.
One of the great selling points of one of my projects is “this is easy to run without needing Docker”.
Is it? At this point I associate Docker with slow build times and later upon layers of abstraction to try and cache and make it not slow. But it’s a never ending battle.
I can’t run Isaac Lab on my work cloud VM because the system has glibc2.34 instead of 2.35. The only way to hack around this is to move everything into Docker. So now there’s a big huge convoluted build process just because I changed a line of code. It’s utterly miserable with no redeeming qualities.
Again none of this is Docker’s fault. But the fact that Linux is so incapable of running software that Docker needs to exist is damning.
For reference Docker is not popular on Windows because Windows can simply run software programs without requiring Docker.
Yet for some reason people think Linux is a good platform for developers. It is bewildering…
I love Go and use it for everything I can but will absolutely concede this point. It's clearly possible to write the most successful container runtimes on earth in Go, but you absolutely have to fight the language/runtime sometimes. Since containers are based on processes, handing off a container spec to a minimal rust runtime seems like what we should all be doing by now. crun has just proven good enough that AFAIK no one has RIIR.
You'll be glad to know of youki then: https://github.com/youki-dev/youki
And implying the system kernel still has to run the container kinda ignores kata containers and gvisor runtimes.
> Did you know that databases are just a file?
oOoOoOoOo
Mostly joking. It's cool to see tech demystified, but I feel like a lot of things we use and take for granted are like this.
Similar to Helm -- in v3 they also removed a daemon (Tiller).
Podman has not required root in years, just saying.
And folders are graphical representations of directories--sometimes, not always--and not directories themselves. When will the newbies get this?
I know a lot about how http works, and it comes in handy, but I can understand how juniors get away with not knowing it as fully as I do.