K3s – Lightweight Kubernetes
k3s.io
k3s.io
The option to use sqlite in place of etcd on an even lighter single node setup makes it super interesting for even lighter weight homelab container environment setups.
I even use it with Longhorn https://longhorn.io/ for shared block storage on the mini cluster.
If anyone uses it with MicroOS, just make sure you switch to kured https://kured.dev/ for the transactional-updates reboot method.
I'd love to compare it against Talos https://www.talos.dev/ but Talos's lack of support for a persistent storage partition (only separate storage device https://github.com/siderolabs/talos/issues/4041 ) really hurts most small home / office usage I'd want to try.
How has your experience been with Longhorn? Performance, flexibility, issues, maintenance...? I'm interested in moving away from a traditional single-node NAS to a cluster of storage servers. Ceph/Rook seem daunting, and I'd prefer something easy to setup and maintain, that's performant, reliable and scales well. Discovering issues once you're fully invested in a storage solution is a nightmare I'd like to avoid. :)
I push it pretty minimally right now, so no great performance testing myself, and I do run it in synchronous mode, so that means its write performance is likely going to be limited to the 1gbps network it syncs over.
It was easy to setup and has been reliably running with very minimal maintanence since.
https://github.com/longhorn/charts/blob/v1.5.x/charts/longho...
My advice for on-prem is to buy storage from a reliable provider with a decent history of hybrid flash/ssd, so that you can take advantage of storage tiering (unless you just want to go all flash, which is a thing if you have money).
If you must use some sort of in-cluster distributed storage solution, I would advise you to exclude members of your control plane from taking part, and I would also dedicate entirely separate drives and volumes for the storage distribution so that normal host workload doesn't impact latency and contention for the distributed storage.
In a professional setting, and depending on scale, I'd probably rely on a storage provider to manage this for me. But since this is for my homelab, I am interested in a DIY solution. As a learning experience, to be sure, but it should also be something that ideally won't cause maintenance headaches.
Keeping separate volumes makes sense. I can picture three tiers: SSDs outside of the distributed storage dedicated to the hosts themselves, SSDs part of distributed storage dedicated to the services running on k3s, and HDDs for the largest volume dedicated to long-term storage, i.e. the NAS part. Eventually I might start moving to SSDs for the NAS as well, but I have a bunch of HDDs currently that I want to reuse, and performance is not critical in this case.
It seems like my volumes are constantly falling into degraded and then rebuilding. Resizing volumes requires taking the workload that's attached down, and then it seems to take forever (15m+) for my clusters to figure out that the pod is gone and a new pod is trying to attach.
Really, it's a PITA and all of the providers' storage classes seem better than Longhorn. Ceph I had less experience with but very similar problems - long-gone pods held a lock on PVCs that had to be manually expunged, or wait for a very long timeout.
I'm working on v2 of my homelab cluster, and I'm going with plain old NFS to a file server with a ZFS pool. Yes, I will have a single node as a point of failure, but with how much pain I've had so far I think I'll be coming out ahead in terms of uptime.
My advice, having done Ceph/Rook, Longhorn, and now Ceph via Proxmox is the latter, assuming you have access to an actual host. Proxmox-managed Ceph is a dream, and exposing it to VMs and then K8s via RBD is easy.
Longhorn is fairly easy to set up, but its performance is terrible in comparison.
Good to know about the performance. This is not critical for my use case, but I guess I'll need to test it out for myself, and see whether it's acceptable.
From the earily days of trying to remember WTF the CRUSH and "rados" acronyms actually meant and focusing on network and storage concerns and [0] ... architectural optimizations [1] ... a RedHat acquisition ... adjusting to the NVMe boom [2] ... and then Rook, probably one of the first k8s operators trying to somehow operate this underwatere beast in a sane manner.
If you are interested in it ... set it up manually once (get the binaries, write config files, generate keyrings, start the monitors, set up OSDs, RGW and you can use FileZilla or any S3 client). For some more productionish usage there's a great all-in-one docker image https://quay.io/repository/ceph/ceph?tab=tags .
[0] oh don't forget to put the intra-cluster chatter on an internal network, or use a virtual interface and traffic shaping to have enough bandwidth for replication and restore, and what to do if PGs are not pairing and OSDs are not coming up]
[1] raw block device support, called BlueStore, which basically creates a GPT partitioned device, with ~4 partitions, stores the object map in LevelDB - and then later RocksDB
[2] SeaStore, using the Seastar framework, SPDK and DPDK, optimize everything for chunky block I/O in userspace
"K8s-native" here implies Rook, which is in no way hands-on for Ceph.
> as long as ... the solution is simple enough to manage without a special degree
Ceph is not simple, that's my point. I assume you've read the docs [0] already; if not, please do so _in their entirety_ before you use it. There are so many ways to go wrong, from hardware (clock skew due to BIOS-level CPU power management, proper PLP on your drives...), to configuration (incorrect PG sizing, inadequate replication and/or EC settings...) and more.
I'm not trying to dissuade you from tackling this, I'm just saying it is in no way easy or simple. Statements like "k8s-native solutions" always make me cringe, because it usually means you want to use an abstraction without understanding the fundamentals.
To be clear, I have read the docs, set it up on my own, and decided I didn't want to try to manage it. I've ran a ZFS pool for a few years on Debian and shifted over to TrueNAS Scale last week; not because I was unable to deal with ZFS' complexity (knock on wood, the only [temporary] data loss I ever had was an incorrect `rm -rf`, and snapshots fixed that), but because of continual NFS issues. I may yet switch back; I don't know - I just no longer had the time to troubleshoot the data layer. Ceph makes ZFS looks like child's play in comparison.
I think you misunderstood me. I'm not inclined to give Ceph/Rook a try because of its complexity, regardless of the state of its documentation. I was refering to Proxmox in my previous comment, in the sense that I don't need a VM/container manager/orchestrator with a pretty UI. If I'm already running k3s, I can manage the infrastructure via the CLI and IaaC, and removing that one layer of abstraction that Proxmox provides is a positive to me. Which is why adding just a storage backend like Longhorn to a k3s cluster seems like the path of least resistance for my use case.
Ultimately, I don't _want_ to deal with the low-level storage details. If I did, I'd probably be managing RAID, ZFS, or even Ceph. For my current NAS I just use a single JBOD server with SnapRAID and MergerFS. This works great for my use case, but I want to have pseudo-HA and better fault tolerance, and experiment with k8s/k3s in the process.
So I'm looking for a k8s-native tool that I can throw a bunch of nodes and disks at, easily configure it to serve a few volumes, and that it gives me somewhat performant, reliable, and hopefully maintenance-free, block or object-level access. Persistent storage has always been a headache in k8s land, but I'm hoping that nowadays such user friendly and capable solutions exist that will allow me to level up my homelab.
Also, I feel obligated to ask “are you me?” at your stated homelab journey. I also went from mergerfs + SnapRAID, albeit shifting to ZFS. I also went with k3s, but opted for k3os which is now dead and thus leaving me needing to shift again (I’m moving to Talos). Finally, everything in my homelab is also in IaC, with VMs buiit by Packer + Ansible, and deployed with Terraform.
Happy to discuss this more at length if you have any questions; my email is in my profile.
To me it seems strange that a systemd unit is used, but I didn't know if I was missing something about the way MicroOS worked.
[0]: https://en.opensuse.org/SDB:K3s_cluster_deployment_on_MicroO...
Nomad seems to be a truly "lightweight Kubernetes" and for the small amount of time I've been using it, it seems to do it's job really well and it's easy to understand all the moving pieces without spending countless of hours reading docs and source code.
Although, it's hard to recommend Nomad for future uses as it sadly stopped being FOSS :/
Are you trying to say that 10 services and 2 nodes would be less on Nomad?
That has not been my experience with k3s. Unlike normal distributions, k3s servers do allow workload to run on them unless you manually disable. From https://docs.k3s.io/datastore/ha-embedded :
" An HA K3s cluster with embedded etcd is composed of:
- Three or more server nodes that will serve the Kubernetes API and run other control plane services, as well as host the embedded etcd datastore.
- Optional: Zero or more agent nodes that are designated to run your apps and servicesSo you can have a full HA k3s cluster with only 3 server nodes and 0 agent(worker) nodes.
> unless you manually change it
I ran into this myself just the other day.
Anyway, lightweight here means that whole bunch of external dependencies have been ripped out. E.g. AWS/GCP/Azure and I believe other things too.
K8s also can do secrets! base64 encoded secrets! But in case you need a more secure solution. Please pull in a Vault helm chart or mess around with sops or whatever. Want to scale out? See how long it will take you until you will have to move your etcd to dedicated hw/vm's. And let me tell you - setting up external etcd in prod sucks. Also maintaining mission control, because there is a bunch of ways to do things and devs love using things that they are not supposed to touch.
Nomad approach is modular this is why it works. Need proper multi dc SD? please run a consul cluster (single binary + single config file)x3. Need mesh networking? same. Need secrets? Vault (single binary + single config file)x3. Pki? same. Or maybe you don't need it? Then you have an option to just not use it.
One thing that i really don't like about it that there is not a lot of docs describing all the ways these tools can interact with each other.
What it actually lacks is versioning, and a dynamic secrets engine, though you could build that with an operator, and things like the postgres-operator do.
You also don't get the whole sealing thing, but I'd argue that's more annoying than useful.
A-fucking-men.
Prior to unsealing, almost no operations are possible with Vault. For example authentication, managing the mount tables, etc. are all not possible. The only possible operations are to unseal the Vault and check the status of the seal.
I am not an expert in this topic, but I read that they added lightweight solutions for both usecases inside nomad in latest version.
Does Nomad expose an API that you can extend with controllers running inside the cluster? Because Kubernetes without operators is not Kubernetes.
Kubernetes operator is a combination of a CRD reconciled by a controller. Without the CRD it's simply a controller.
I do like the linked post, it's well written and a valiant effort, but not close to comparable.
Alright, so, what is it, then? I've been unfortunate to work at firms that generally have a minimal level of competency with Kubernetes, but across several billion dollars worth of firms, not a one has used operators in any capacity, but they leverage Kubernetes substantially. Help me understand the gap, would you?
Among its many strengths, core services for running Kubernetes running inside Kubernetes itself is one of its greatest.
> it's easy to understand all the moving pieces
It's legit to leverage your pre-existing knowledge. Nomad + Linux Pets + Other Hashichorp Pets works well.
There are many ways to run an application. Kubernetes + Machine Cattle, in my experience having started from zero, is superior to any other application deployment paradigm I've ever used.
My personal experience is that it is very, very hard to find a job right now if your professional experience is primarily non-k8s orchestration systems. Most job positions out there require deep Kubernetes knowledge as well as hands-on experience with different supporting products like ArgoCD and Flux.
I chose Nomad for a large initiative at my current employer and it is honestly pretty amazing given the use case but I regret choosing it because I feel like I'm unhirable now given that every devops/SRE/platform engineering position open on the market (each one with hundreds of applicants) is heavily Kubernetes focused.
I only recently had to use AWS EKS, and am pretty sure that many people dislike k8s because that's the main incarnation of it they experience (how is there no pre-installed ingress, and even the setup for their own load balancer is a 10+ step manual process?).
And if you want to use EKS-A (EKS-Anywhere) it's even more effort to do the AWS IRSA setup.
K8s is complete overkill at best for the majority of companies/workloads and introduces lots of other dependencies on teams in terms of workflows and architectures to make things not be a dumpster fire for all but very mature teams.
On the professional side, k8s handles orchestration issues far better than anything else I’ve worked with. Using autoscaling instances is a nightmare by comparison, and requires most of the same initial effort to do it right.
There is probably a middle ground where it requires a certain amount of complexity in the K8s configuration that isn’t worth it compared to other platforms (especially on bare metal), but I haven’t found it, yet.
I think it’s easy to forget how much goes into running even a simple application with high uptime requirements. Kelsey Hightower makes some great points about this here:
k3s requirements: 1 node, 512MB RAM, 1 CPU core
Nomad requirements: "Nomad servers may need to be run on large machine instances. We suggest having between 4-8+ cores, 16-32 GB+ of memory, 40-80 GB+ of fast disk and significant network bandwidth."
https://docs.k3s.io/installation/requirements
https://developer.hashicorp.com/nomad/docs/install/productio...
The licence change impacts only companies like Digger or Spacelift that build their platform abusing TF oss offering.
Why is „AWS attacks OSS” bad and „Spacelift attacks OSS” good ?
Its exactly the same case just size of companies is different.
The idea is to more easily get going with Kubernetes, not to provide an alternative. Nomad and K3s serve a completely different use case.
You're free to run either on one node, but other than for toy purposes like demos and learning, why use a cluster compute system when you're not actually running a cluster?
[0] https://www.suse.com/news/suse-completes-rancher-acquisition...
[1] https://rancherdesktop.io/
Disclosure, I work for SUSE and started Rancher Desktop.
So many questions...
Rancher Desktop will provide the CLI tools to work with it.
The idea is "it just works". It's a desktop app so it installs what you need.
We use it as a free replacement for Docker Desktop and it works very well.
Unfortunately, we cannot use the Kubernetes part of it because it comes with "steve" (used for dashboards), which uses and blocks (!) the same ports as our application, and those ports are not configurable. So, please please solve this request here: https://github.com/rancher-sandbox/rancher-desktop/issues/18...
* Traefik daemonset as a load balancer
* Helm controller that lets you apply helm manifests without the helm command line
* Upgrade controller
* Sqlite as the default backing store for the k8s API
* Their own local storage provisioner
K0S has a lot of the same goals: be light weight and self contained in a single binary. But K0S tries to be as vanilla as possible.
Choosing between the two it comes down to your use case. Do you want light weight and compatible (k0s), or lightweight and convenient (k3s)?
Edit: formatting
* A helm controller is included in k0s
* Etcd is bundled and bootstrapped automatically which I perfer because I dont want the overhead of the translation that Kine does. Although Kine is available for a non-etcd datastore if that is preferred.
* Upgrade controller is included (autopilot).
* They have a local storage provider based on OpenEBS.
* Ingress is missing, but due to the built in helm controller that can be boot strapped upon cluster initialisation.
Overall, together with k0sctl and its declarative configuration it is easier to deploy k0s than it was k3s.
I don't have specific numbers unfortunately since it was years ago I benchmarked Kine against etcd. But I had a better results with etcd both in cluster and single node.
I happened to stumble upon this paper that echos my experience. https://programming-group.com/assets/pdf/papers/2023_Lightwe... Particularly, the high controller cpu usage (even for an empty cluster), and higher latencies.
my problem with etcd was very high and constant I/O and CPU usage. I don't mind the latency.
Lots of teams are using K3s to run Kubernetes at the edge and in IoT applications, and with good reason. It's a fantastic Kubernetes distribution that's well-maintained, easy to get going with and well-documented.
(Ironically, if you look at the first commits to kubernetes/kubernetes, the Kubernetes components were originally shipped as a single binary. They decided to break them up later to simplify releasing, but the k3s monolith lives on.)
It also has Traefik set up with sane defaults, and the local path provisioner is also pretty good, too. But recently I've moved to Longhorn since I plan to eventually scale past 1 node. My only complaint about Longhorn is that applications that are write-heavy and delete old data (e.g. Prometheus with short retention) will require aggressive trimming (e.g. trim once a day) to keep the actual size of volumes down. Besides that, Longhorn makes backups to S3-compatible storage very effortless and you get RWX volumes, too!
Regarding k3s itself, you can persist modifications to the way some components (e.g. Traefik) are installed through a HelmChartConfig CRD. This is what I personally use so I can use Traefik to route SSH traffic for Forgejo. Another nice thing is that although components like kube-proxy are baked in to the single k3s binary, you can still scrape metrics with Prometheus provided that you expose their endpoints somewhere on your cluster network.
https://github.com/rcarmo/azure-k3s-cluster
(I tweak this every now and then since I both used it as a training sample for my customers/peers and as a way to run my own batch processes as cheap as possible)
Im am completely oblivious to how k8s works.
Kubernetes has become an interface, independent of implementation.
It's much like how POSIX is a standard and there are many implementations.
And yes, POSIX has a learning curve too, but I'd rather learn that than anything proprietary, or non standard/rapidly changing.
But let's say you have many computers and want to put them to work. Suddenly, kubernetes becomes the simplest of all options.
> Everything should be made as simple as possible, but not simpler.
- Einstein
You can use something like Ansible to make it a bit easier, if that even makes sense in your use case.
I connected motion sensors to battery & wifi enabled RPis, built a remote circuit to control the fog machine, and ran a k3s cluster with NATS to bring it all together. 10/10 would do again.
Does anyone have experience or recommendations in this area? I've heard about a few tools but wanted to gather insights from this knowledgeable community.
I switched to it completely, it’s very convenient to have both fast (-est on Mac) Docker support and a really smooth VM setup for running occasional Linux tools (such as Yocto in my case).
Edit: added some background info to my recommendation.
You can also use k3s; it's hella easy to get started with and it works great.
Extremely quick to stand up a single node cluster, or many types of VMs in lima.
https://github.com/lima-vm/lima
limactl start template://k3s
colima start --kubernetes
hetzner-k3s : This is a CLI tool to quickly create and manage Kubernetes clusters in Hetzner Cloud using the lightweight Kubernetes distribution k3s from Rancher. https://github.com/vitobotta/hetzner-k3s
Kubernetes on Hetzner Cloud the easiest way https://vitobotta.com/2023/01/07/kubernetes-on-hetzner-cloud...
If one willing to have in-house k8s today I would recommend https://deckhouse.io/ (I'm not affiliated with them)
Agreed:( Someday I'd like to see how hard it would be to use microos to build a replacement cheaply (cheap in terms of maintenance; ideally it would just be a combustion config or so that deployed k3s on vanilla microos)
> And while k3s sounds easy, it's not after even a slightly larger scale.
How does it fall apart?
Agent nodes failing to join after reboots or long network partitioning. Manual re-joining works. Additionally, hard to debug issues with the supplied Traefic.
There's this issue that goes back to 2019 that got closed without any real resolution. I moved on but I'm curious if anyone else has measured this?
It’s fine to use docker compose if it does what you need. If you start hacking weird things using Docker compose, it’s perhaps time to suffer and learn kubernetes.
How do you migrate services and storage from one cluster to another?
You have to think of k8s as more of a specification of what can be done with multiple implementations. k3s, kind, microk8s, minikube are all implementations of kubernetes that you might want to try out.
k8s is much like the original IBM pc or a linux distribution. There are a bunch of interchangeable parts that you may land on depending on your needs.
Why use this over something else? I've found k3s to be really simple to setup and upgrade, which is why I use it for my home lab.
k3s, at least, supports multiple controller nodes on setup.
Where things really get squirrelly, at least with k3s, is storage and ingresses. k3s supports both but in a super robust fashion.
If you are in a datacenter, it's best to use the K8s distributed by that datacenter (EKS for AWS, for example).
The EKS provided by a datacenter will have integrations with external to Kubernetes resources you will almost certainly want (like storage, DNS records, or load balancers).
See here: https://docs.k3s.io/
It's fully compliant but opinionated towards the smaller side.
K3s is the lightweight distro. You can install and run it from a single binary on pretty much any Linux distro.
half(kubernetes) == 5 -> k3s
> We wanted an installation of Kubernetes that was half the size in terms of memory footprint. Kubernetes is a 10-letter word stylized as K8s. So something half as big as Kubernetes would be a 5-letter word stylized as K3s. There is no long form of K3s and no official pronunciation.
If you need/want to serve v6 clients, put a v6 load balancer in front. You're likely going to need that lb anyway.
ipv6 and routable ips is way simpler than patching layer 2 networks together.
https://www.jeffgeerling.com/blog/2022/quick-hello-world-htt...
Either you use it a lot (likely at work) and are therefore comfortable using it outside of work as your one-size-fits-all solution to everything, or else you don't use it a lot and see it as a complexity behemoth that hides all of your system behind esoteric CLIs and config files that just make everything a nightmare to manage.
I've very recently moved over to a home k3s cluster--a couple old desktops and some relatively new ARM SBCs with NVMe slots. I don't use it at work, at least not directly, but it's been the least-painful solution I've found to run stuff; once you understand the model (not trivial but not crazy) and find the headspace to explain why things in k8s are the way they are, reasoning through it is pretty easy (which is really just the story of any moderately complex system, exacerbated in a few ways because of some Google-attitude carryover), and I can also yank half the machines out of the cluster before an application fails.
(Both Swarm and Nomad are fine tools, but the lack of easy resources to get up to speed led to me failing them out of consideration.)
In the context of local or home servers: my raspberry pi currently has like a 3 year uptime now, but if it randomly croaked I'd just buy a new raspberry pi, docker pull the image, and get it up and running again.
So you are saying the main benefit of k8s is that if some of your hardware in the cluster dies then your services are automatically still up and running? Isn't a home k8s setup vulnerable to the same failure as well if the master node gets smoked?
Seems like a lot of complexity cost for something that happens extremely rarely in a home setup at least (I could see how it would be useful for high reliability cloud systems).
The benefit to me is if one of my nodes physically dies i have enough room on my cluster to still run all my pods, they just automagically reschedule to another node.
If my master dies then I have to do a little extra work, but not much.
- I have three control planes, not one, for about 20 x86-64 and 24 arm64 cores and ~160GB of RAM across 7 nodes. For much the same reason as my NAS uses raidz2, I don't have to drop everything and fix the universe if something fails because Home Assistant just keeps going.
- With Longhorn, I have three instances of every data volume in the cluster. Longhorn also provides incremental data backups to an S3 server--Minio on the NAS, in my case, which replicates to an offsite.
- I can run things like CrunchyData's Postgres operator, which provides a solid baseline setup and additional features like point-in-time restore--and as I run personal but public-facing apps on the cluster this is beneficial.
- Logging and monitoring are centralized and were easy to do. I don't have to go attach to a container, I just go look at Grafana.
- I have a consistent interface for working with all my applications, and I like it more than Swarm's by a lot.
By having this cluster, made out of spare computers, some cheap NVMe drives, and a couple SBCs I wanted to experiment with anyway,I've removed everything I care personally about from cloud providers except for offsite backup and my email sending. I know where it is, I can hack on projects trivially, and I know it'll be there tomorrow so long as the house doesn't burn down.
And I didn't learn any of k8s at work.
Opinions are fine and you may as well hate K8s or any tech being discussed. I just hope to see less of "people who like this just because they don't know any better vs the smart ones like me who know The Truth™" comments.
I'm trying to grapple with the fact that k8s is apparently ultra popular technology, yet every time I try to read up on it or start using it, it quickly turns into a suck fest in terms of learning curve and complexity. So my conclusion is that the people using it "for fun" must have acquired a taste for it first by using it all day at work and I wanted to see if anyone who acquired the taste might chime in as to what made them see the light.
-can run any workload
-on any hardware
-at any scale
-with load balancers
-and persistent disks
-and auto scaling
-and resource limits
-and certificate management
-and no single point of failure
-and etc.