Linux containers in a few lines of code
zserge.com
zserge.com
isolation layer: the piece that limits privileges and resource usage. (On linux, this is usually handled by cgroups and the kernel, but could also be handled by something like kvm for vm-based containers)
raw container configuration: Given an image and some metadata (like cpu limits), launch an isolated process. (On linux, this is usually handled by runc when working with cgroups)
container api daemon: Manage the list of container processes and available images. Provide a unix socket based API for manipulating isolated processes, launching, deleting, connecting, etc. (In the case of docker, they provide a daemon which abstracts the containerd daemon, or you can use containerd alone without docker)
container command line tool: Provide a user/developer interface to the three things above. This is the docker command. When you install containerd without docker this is the ctr command.
Docker, which is probably the most famous container distribution, pairs the docker command with the docker daemon to abstract away the containerd daemon, runc, and cgroups.
If you use containerd alone, you get ctr/containerd/runc/cgroups.
There's a standalone command line tool (crictl) which replaces both ctr and docker and can be used on top of either the docker daemon or containerd.
[1] Container systems seem to have a relatively complex abstraction over what is a relatively simple architecture.
Being able to run containers like any other process and leave the lifecycle management to systemd is actually really nice.
When I went to work at Red Hat I decided to really try Podman, and I love it now. Once I discovered that Podman supports Kubernetes Pods (same YAML and all) I realized how clunky docker-compose actually is. Since most of my projects now run in K8s anyway, it's awesome to have the app run as a Pod locally so it can easily be tested with any sidecars in the same configuration.
There are still uses for docker-compose, and I don't expect it to disappear from my life completely any time soon, but Podman does have a great place in my toolkit.
With stuff like RootImage and the various isolation settings you can have a configurably sandboxed container right in systemd.
Or just use systemd-nspawn if you want it more preconfigured.
* the combination of all technologies touched here in the example, but it's missing a very important-one: the layered filesystem
* A defined packaging format and way of distributing images
* Having an API to talk to these things easily.
That is what kickstarted and revolutionised the container space, but there were still many technology gaps and questions. They tried to put everything in one single tool, which was a double-edged sword I think: it made usage easy which accelerated adoption, but also caused it being a big 'monolith' project, which is a bit ironic given that it ended up being extremely suitable for micro-service architectures.
For me as an early adopter, it was very clear from the start that the main problem was how do you actually deploy this in production. Docker saw a very large part of the conceptual bigger picture, but they had any clue how to address this, but to be fair, very few did.
Right now, a lot of lessons have been learned, and standards have been created in the container space.
Nowadays, for production loads, k8s has become the de-facto standard container api, and Docker itself doesn't really carry that much weight anymore, the runtime itself isn't important anymore. It being podman, runc, or still a full-blown docker? I couldn't really care less. Docker's stronghold will remain the development environment for a while though, just due to the massive amount of resources and easy to use solutions out-there...
Clarification: cgroups only control access to compute resources, like CPU time and memory. To control privileges, e.g. access to the filesystem or network, you need namespaces, which are a completely separate kernel feature.
Yep, it's really not rocket science.
I find that very useful and usable resource isolation (i.e. most of what I actually use containers for) can be achieved with processes + cgroups + chroot.
Unfortunately networking complicates things since DNS doesn't return ports, only IP addresses. A local DNS service that returned IP:port (and appropriate connection APIs at the application level) could eliminate much of the need for network namespaces. Putting services behind front-end gateways also works (indeed it's usually the model for web services.)
(Disclaimer: I actually do use network namespaces frequently and have fun doing so, but I tend to think that networking APIs are too low-level.)
DNS can return IP:Port pairs, using SRV records instead of A/AAAA records, but for some reason it never took off :( I guess because these days everything uses HTTP as the transport layer, and HTTP doesn’t support it?
I’ve personally been getting in to writing minimal code because I’ve grown frustrated with how simple tasks can result in complex code that’s difficult to maintain. Minimal code is easier to come back to. A 100 line script can be easier to understand than a 500 line script.
> system("ip link add veth0 type veth peer name veth1");
Always, always, always use exec*() APIs.
[0] https://github.com/p8952/bocker
[1] https://github.com/p8952/bocker/blob/master/bocker#L61-L90
Definitely worth the $12.
You can download it for free too: https://gumroad.com/l/containers-zine/buyonegiveone
There is an official option to either pay or get it for free. Do support her if you can.
For me this is enough to get a container running:
lxd init
lxc launch ubuntu mycontainer
lxc exec mycontainer bash
When I think about containers, I think about isolated system images, lightweight VMs, which I can use and adapt to solve a problem at hand.
When I hear Docker, I think about static, locked-down application-images made by others, to deploy in the cloud, which I'm not given to adapt to my needs, and I also think about things which is not natively integrated into my Linux-distro and for which I will have to provide glue manually. I also think about a startling amount of complexity in a new stack which I would have to learn, just to manage what is actually really just basic Linux-systems.
While all that may or may not be true, that's what I think about when I hear "Docker".
Hearing "containers" on the other hand, makes me happy. And yeah, I'll got with LXC/LXD any day.
after that read more about overlay, overlay fs, for historical reasons aufs.
that being said, the attack surface is wider than say if you would run it in a VM or its own physical machine
I'm asking about the code in the webpage, not code I'm writing personally. I'm saying let's assume it has error-handling added to it. That's it. I am not writing any code otherwise. Is that code doing "the right thing"? Or are there more things it needs to be doing?
Emulating an entire machine, all of its diverse hardware, their bespoke protocols, and all the weirdness of the x86/amd64 ISA is fraught with peril. It is a large attack space. So too is the Linux kernel.
And frankly, inside the VM, half of us are running Linux anyways. I feel like for a lot of use cases, compromising the VM's guest OS (Linux) is enough to have a really bad day. Compromising the hypervisor? Bad, yes, but now it's AWS's problem.
There's more to containers, too, than just the security thing, and I think there are enough other advantages (I can more easily bin-pack services together; I can separate the FS and thus dependencies of unrelated components; I can more easily test them; etc.) that containers are worth it. Often and even on top of a VM. (My current work is with containers, and we run them on VMs.)
This is only possible in this example because the container has the full capability set (including CAP_MKNOD) and the devices cgroup hasn't been configured to restrict device access. Real container runtimes always restrict device creation by default, and usually don't allow CAP_MKNOD by default.
As another comment suggested, user namespaces are another hardening feature, but not all container runtimes enable it by default. Podman does, Docker doesn't. In fact user namespaces are so powerful that I believe they pretty much cover most of the hardening provided by the three features I listed above. If you're wondering why they're not enabled by default in Docker, take a look at this [1].
Exploiting the missing isolation mechanisms, the following bash commands will allow you to escape from the author's containers:
$ ls -al /sys/dev/block # find the root fs device (e.g. /dev/sda1) major and minor device numbers (e.g. {maj=8, min=1}, {maj=259, min=1})
$ mknod --mode 0600 /dev/host_fs_dev b $major $minor
$ mkdir /host_fs && mount /dev/host_fs_dev /host_fs
(warning: shameless plug to my posts follows:)
If you want more details, I wrote a post on this exact same problem in the context of three vulnerabilities I found in rkt (another container runtime) [2].
Beside the issues above, the author's runtime also exposes host file descriptors like /proc/self/exe that can be used to escape the container. This is a post I wrote on runC CVE-2019-5736 that explains this kind of issues.
[1] https://docs.docker.com/engine/security/userns-remap/#user-n... [2] https://unit42.paloaltonetworks.com/breaking-out-of-coresos-... [3] https://unit42.paloaltonetworks.com/breaking-docker-via-runc...
Could you please explain/point me to some information/source, why docker can't use -net=host namespace if userns is enabled, while on the other hand rootlesskit[1] which uses userns by default, dont have problem with using host netns (--net=host) ?
Linux containers in 500 lines of code (2016) https://news.ycombinator.com/item?id=22232705
https://www.polarsparc.com/xhtml/Containers-1.html https://www.polarsparc.com/xhtml/Containers-2.html
A similar snippet[1] exists for go, and it doesn't do anything particularly special either.
I don't know, maybe David beazley has altered my sense of what "from scratch" means.
[1] https://gist.github.com/lizrice/a5ef4d175fd0cd3491c7e8d71682...
Like here's David's "Build Your Own Async" [1], which I prefer over Philip's (still extremely good) "What the heck is the event loop anyway?" [2].
It's one thing to tell how something works, but to successfully show what the hell its actually doing under the covers, just conveys much more information.
[1] https://www.youtube.com/watch?v=Y4Gt3Xjd7G8 [2] https://www.youtube.com/watch?v=8aGhZQkoFbQ
$ echo $$ > tasks $ mkdir /dev/cpuset
$ mount -t cpuset cpuset /dev/cpuset
$ cd /dev/cpuset
$ mkdir Charlie
$ cd Charlie
$ /bin/echo 2-3 > cpuset.cpus
$ /bin/echo 1 > cpuset.mems
$ /bin/echo $$ > tasks
# The current shell is now running in cpuset Charlie
# The next line should display '/Charlie'
$ cat /proc/self/cpuset
Otherwise you'd have to argue that configuring rlimits actually makes your shell a container, which is too much of a stretch (for me at least).
rlimit is sort of a thing but it's not actually effective so to me it's not part of the picture. If unix limits worked, Google would not have needed to contribute cgroups before deploying Borg. Indeed, in this LWN article which is actually about control groups, they call control groups "containers". Just shows there is not a universal meaning of the term.
The earliest control groups patch I can find says "We use the term container to indicate a structure against which we track and charge utilization of system resources like memory, tasks etc for a workload." It doesn't say anything about isolation, namespaces, or security, but it uses the term container to describe resource control.
> Just shows there is not a universal meaning of the term.
Yes, there is. That meaning has just evolved since 2007.
pivot_root() doesn't suffer from this problem because it applies to the entire mount namespace, and thus its implementation could be made much safer. Instead of just changing the current process's filesystem root, the actual / mount of the mount namespace is swapped with another mountpoint on the system (and the old root is mounted elsewhere). Thus once the old root is unmounted there isn't a way to get back to the old mountpoints. This isn't perfect protection (magic-links and other mounts could expose the host filesystem) but it is a damn sight better than chroot(). Oh, and nesting pivot_root()s doesn't cause a breakout.
Note that this different behaviour in relation to mounts has resulted in completely unrelated security bugs with containers (such as being able to bypass procfs masks because chroot() doesn't hide the unmasked procfs in the host mount namespace). This is why us container runtime authors always tell people they should never use chroot() and always use pivot_root() -- though sadly sometimes chroot() is needed because pivot_root() doesn't work on initramfs.
(I'm one of the maintainers of runc, the runtime which underlies Docker/podman/containerd/cri-o/...)
Are people actually attempting to boot a super-minimalist system that just has a kernel and an initramfs with something like docker into it where they don't bother with a rootfs at all and just start running containers directly from the initramfs? That's kinda cool, if that's the case.
This means that setups where you have the entire OS image in initramfs, and you try to run a container (even if it has a different filesystem as its rootfs) it will fail with pivot_root(). There are solutions for this but they require changing how the system is started (which can be a bit complicated depending on what system you're using to build your initramfs). From memory, minikube has used --no-pivot-root for a while precisely for this reason, though I believe they have switched away from it sometime recently.
Could the mods consider renaming the submission?