Comment by jsmith45 | Hacker News Reader

Comment by jsmith45 | Hacker News Reader

Parent Full thread

"docker containers" are more accurately called OCI containers, and have been standardized so that various container runtimes can use exactly the same container images.

Kubernetes can use docker runtime (dockerd) to run OCI containers, but Docker Inc strongly discourages the docker runtime being used directly for infrastructure. Docker runtime imposes a lot of opinionated defaults on containers that are often unwanted by infrastructure projects. (For example docker will automatically edit the /etc/hosts file in containers, in a way that makes little sense for Kubernetes, so Kubernetes has to implement a silly work around to avoid this.)

Instead Docker Inc recommends using containerd as the runtime. containerd implements downloading, unpacking, creating CRI manifests, and running the resulting containers all without implementing docker's opinionated defaults on top. Docker itself uses containerd to actually run the containers, and plans to remove it downloading code in favor of using the one from containerd too.

The only advantage to using docker proper for infrastructure projects is that you can use the docker cli for introspection and debugging. Kubernetes has created its own very similar cli that works with all supported backend runtimes, and also can include relevant Kubernetes specific information in outputs.

> But Docker Inc strongly discourages the docker runtime being used directly for infrastructure.

Is there a list of these defaults or other downsides to using docker instead of containerd?

Docker did a nice blog post on this a few years ago. Docker uses containerd for running containers. It just does things on top of it that you don't need with Kubernetes. There's a nice diagram in the post, too.

https://www.docker.com/blog/what-is-containerd-runtime/

I'm not sure of any such list, but using containerd directly is faster, less likely to break k8s when docker adds new features, etc.

Much of this all stems from the flak infrastructure people gave docker when they made swarm part of the engine. But it comes to more than that. Docker has its own take on networking, on volumes, on service discovery, etc. If you are trying to use docker as a component of your own product, at least some of these are likely things you want to implement differently. And the same may well be true of any new features docker wants to add in the future. At which point one must ask why bother using docker directly?

containerd was quite literally created when docker decided to extract the parts of docker that projects like kubernetes might want to use. It has evolved heavily since then, but that really does capture the level at which it sits. This leaves dockerd in charge of things like swarm, docker's view on how networking should work, docker's take on service discovery, dockers view on how shares storage should work, building containers, etc.

Half-OT: What are alternative runtimes and why would you use them?

For a runtime in your Kubernetes cluster there are containerd and cri-o. These are good for Docker / Open Container Initiative images.

There are others... some for non-Docker image support. There are people running other things than just Docker these days. They are more niche case.

In the OpenShift world we use CRI-O and it has been awesome for us. I've never actually had it be a problem. Occasionally have to SSH into a node and inspect with crictl to see what's going on but it's almost always PEBKAC that points at CRI-O when it's not CRI-O's fault. I'd definitely recommend looking at it.

How has been your experience with crictl ?

Does it do everything that docker cli does ? Build, pull, etc ?

Only docker, buildkit, podman, buildah, and some non-Dockerfile based tools are able to build. CRI-O or containerd do not have build functionality. It does have pretty much everything else you expect like pull, ps, run exec, attach, rm, rmi, etc.

If you need builds, I'd suggest either run dockerd alongside a containerd kubelet, or use buildkit.

That's super valuable. Thanks! I did not know about buildkit (and buildctl), but now see it.

Very valuable. Thanks!!!

The main alternative runtime that I know of (at the level of containerd) is CRI-O. These runtimes are at the level of fetching images, preparing manifests etc. I'm not really sure what benefits they provide. CRI-O is intended to be kubernetes specific, and thus lacks any features that containerd would have that k8s does not need. This in theory ought to mean smaller, lighter, more easily auditable code.

There is another lower level of runtime, the OCI runtime, of which the main implementation is runc. Alternatives have interesting attributes, like `runv` running containers in VMs with their own kernel to get even grater isolation, `runhcs` which is the OCI runtime for running windows containers, etc. Most if not all of the higher level runtimes allow switching out the OCI runtime, but in general sticking with the default of `runc` is fine.

Yeah, the terminology around "runtime" is confusing and is used inconsistently. As you say, the actual runtime is something like runc which CRI-O (and I believe containerd) normally uses. CRI-O, as the name suggests, is an implementation of the Kubernetes Container Runtime Interface--which should work with any OCI-compliant runtime.

One such alternative runtime with tangible differences to containerd/cri-o is Kata[1], which actually runs OCI images as microvm's. This has some benefits if the applications you're running are untrusted/need additional sandboxing, such as if you're running a PaaS on bare-metal k8s.

[1]: https://github.com/kata-containers/runtime

I did some research, and it also seems that there is a way for containerd to manage Firecracker microvms.

A simple example would be mostly running Linux applications on Linux machines (= containers), but having a Windows application that needs to run in a Windows VM on a Linux machine (VM, or a container in a VM).

OCI container images and Docker images aren't the same.

it would be more accurate that they both conform to the OCI runtime spec, right?

Why do you spell containerd as "ContainerD"?

You wrote dockerd without caps.

dockerd is the literal name of a binary while containerd is the name of a project. As far as I can tell containerd stylizes its name in all lowercase but more than half the time I still see it written like a standard name, ContainerD, exactly like this.

Being nitpicky here, but the canonical representation of "containerd" is all lowercase, as in the logo.

Reply on news.ycombinator.com