Talos: OS for Kubernetes

I wish these sales-type pages had a "click here if you're an engineer." The page would contain a buzzword-free, straightforward explanation of what the product's purpose is, what its major features are, and what interfaces are exposed for control and flow.

I have passed on countless products when evaluating solutions for having very flashy sales pages with no actual description of the product itself. "Improve your ROI! Decrease Downtime!" aren't features, per se, but potential benefits that can only be determined if the product is a fit, and I can only determine fit if I know that the hell it does.

/rant

Linux is already an OS.

We are working with a number of users currently. Please see our README for community meeting times if you'd like to chat some more!

The whole point of Kubernetes is that you don't think this way. Replacing a node is not an impactful event if you're using K8S correctly.

We are taking two approaches to this. The first is that you could roll out a replacement node and shutdown the old one. In bare metal scenarios this is much harder so we implemented in place upgrades, but they work very similar to creating a new node. Since Talos is immutable and runs from RAM, an in place upgrade consists of shutting down all services, and then wiping the disk and performing a fresh install. We then reboot the node and its as if you wiped the machine clean and installed the new version of Talos from the get go. This is all via the API by the way.

Yes we do. I personally am working on the channel based approach for our 0.3 release that we just started developing. I would love it if you could make a meeting some time soon to chat some more. User feedback will be really help.

Since Talos is built entirely in containers and we control the entire toolchain, I believe you could achieve the same with Talos.

We do support storage volumes. A recent change in Rook seems to have broken how it works with Talos, but we know storage is important and we are working on fixes. We would love to land support for Nvidia GPU containers. Your not the first to ask for GPU support, so I'm certain we will be taking a closer look at that.

Interesting. We PXE boot Talos in packet. One user is even go so far as to PXE boot Talos on every boot.

I spent a lot of time in v0.2 building Talos for ARM, and it works, but there is a good amount of work to be done to make it official. We need to setup ARM nodes to run our builds from, and refactor our build logic to account for multiple architectures. There is also a bit of work to be done around the boot loader logic since we use syslinux. We are close, just needs a little push.

I haven't tested in Virtualbox yet. Would you mind either join our slack or creating a GitHub issue where we could provide better support?

Thanks! I haven't heard of Zulip, I will be taking a look!

First of all, thank you for taking the time to write this out. The feedback is very valuable. I will do my best to address each comment.

Let me start by laying out our design constraints. We knew we wanted a handful of simple features:

- minimal

- immutable

- and secure

and we approached them with the willingness to do whatever it took to achieve them, no matter how different it would be from any Linux distribution today.

The degree to which we want to obtain minimalism is what I like to call "ultra". Not a single file should be on the image that isn't absolutely needed. Furthermore, not a single process should be allowed to run that isn't required to obtain the goal of running Kubernetes. So we started by creating an image with just enough to run the kubelet and the kubelet only. Obviously, this isn't practical, but it was a place to start.

In implementing our immutability design constraint we decided to:

- make the root filesystem read-only - have no package manager - not allow any generic use of the OS (i.e. it would be only for the purposes of running Kubernetes)

When optimizing for one thing, you often degrade another. In our case, if we optimize for minimalism, then immutability becomes degraded. We need to address a way to manage and debug the node, and we need libraries/binaries to do so. With no package manager, this means everything must be baked into the image, and thus we degrade immutability.

Tacking on yet another design constraint, security, things become even more interesting. The more you add to a system, the higher the risk in vulnerabilities. The more allowed permissions in a system, the higher the risk in vulnerabilities. So minimalism, and immutability actually complement security. In our case, security has the highest priority of all, which means we aren't willing to degrade anything that supports the security of the system. So minimalism, and immutability must be present.

Aside from our design constraints of minimalism, and immutability, we also avoid C as much as possible. We want to build something using a modern language for all the reasons you would choose a modern language over C today, but mostly for security purposes.

Taking all the above into consideration, this meant that we are still left with figuring out how to manage a machine without degrading minimalism, immutability, and security. So without tooling on the rootfs, without a package manager, and without a way to run custom processes, we still need a way to obtain the information we need from a machine. Thus the API was born.

The API doesn't only solve the management issue, it also reenforces all of our design constraints:

- we can keep the image minimal with a single binary serving the API - we can keep the image immutable by building a robust API - we could retain security by using mutual TLS and offering a read-only API - we could write it in a modern language, using modern tooling (golang and gRPC)

At this point what need is there in SSH/console access if the design constraints essentially remove all usefulness in console access? The problem isn't necessarily the need for SSH/console, its the need for a way to get the data to make informed decisions.

There are also additional benefits to an API. There is a reason the concept exists. With an API you get a standarization, strong types, and constistent and well known output formats. The benefits are many.

I'd like to also point you in the direction of an execellent talk given this year at Blackhat: https://swagitda.com/speaking/us-19-Shortridge-Forsgren-Cont.... The section on D.I.E. in particular will add some additional support to the reasons I gave above.

That is my lengthy response to the reasoning behind the removal of SSH. Remember, just because we don't have SSH baked in, nothing is stopping you from running a DaemonSet that has SSH.

As for a custom kernel, we would love to support this. Happy to take in feedback here. We create Talos in containers and our goal is to create the necessary tooling to make this dead simple.

As for node joins, they do not happen with the trustd username and password. We use kubeadm under the hood, so its token based, and possible to have a TTL. We have since moved to token based approach for trustd as well. Note that the trustd token simply gives the node the ability to a worker to request a certficate for OSD, so that you can hit the node's API.

We are currently working on an upgrade operator and it is planned for v0.3. If you would like to have some say in the direction we go, we would be happy to have you in our community meetings!

You make good points about the diagrams. It is clear from this post that we have work to do around the documentation.

And finally, Talos is not based on any distribution. We have a toolchain that we build, and subsequently build our entire distribution from.

I hope I have answered your questions well enough. I look forward to hearing back from you. Your input is valued, and we really would like to use it to turn this into somethi great!

What if it's mainly but not only k8s that you want installed - wireguard for inter-node networking for example, or something to support GPUs/other hardware?

Thanks for the clarification, I can see the use.

this was not clear (to me anyway) from the docs.

FYI: Talos is a trademarked name. In tech. So is Telos. This will be very confusing and likely cause you issues

Installing Kubernetes has gotten easier since the time you last looked at it then. Having used Kubeadm and then RKE from Rancher, I can get a 3 node (non-HA) cluster up in under 10 minutes. This is on regular RHEL 7.4 machines when I last did this in a non-automated way.

That's great! Keen to see your threat model and architecture.

Yes, I read the code.

I am not sure that you want to call it an OS per se, since kubernetes itself is the OS (manage memory, schedule processes, etc).

So there is no default CAPI provider?