My IRC client runs on Kubernetes
xeiaso.net
xeiaso.net
"simple/complex" is not the right paradigm. The real SRE controversy is "unique/standard". Yes, the standard approach is more complex. But it is better _in practice_ to have a single approach, rather than many individually-simpler, but in-aggregate-more-complex approaches.
Kubernetes is never the perfect solution to an engineering problem, but it is almost always the most pragmatic solution to a business problem for a business with many such problems.
Your problems now include k8s networking as well as VPC networking.
Until you need to update a change or upgrade something or patch a security issue. The most dangerous code is code that doesn't run often. Like backup restores. Ticking time bombs unless you're really careful.
> Bringing k8s into the mix is a whole other application layer that's providing services to your code. Things like sgs for pods, cni plugins, coredns, is code that wouldn't be there in a non k8s world.
The same applies to everything running around your tech stack that you didn't write. That includes the language runtime, the compiler, the operating system, all OS libraries you use, and the libraries your app imports. The ship has sailed and K8S isn't much of an addition.
I haven't hit a CNI bug in probably 5 years and that was only on Amazon's horrible CNI code. I've hit a ton of bugs in my software stack. And the libraries I depend on. And my deployment stack. And my cross-service networking stack at the application layer.
If k8s were as complex as it was, but didn’t force that complexity onto the user, I’d agree.
I can understand that it doesn't come off that way but I actually like k8s and the surrounding ecosystem. But deploying k8s makes your infrastructure as complicated as k8s itself, which can be a massive win because it works in both directions.
I believe the OS comparison is the most apt, both in the sense of the layer it represents, i.e. abstracting away physical machines and managing resources for applications, but also from what it means: one used to have to rewrite a program for each new computer as they were all unique (incl. CPU), just like each corp had its own way of managing infrastructure, clusters of nodes and automation.
Any new application needed to be integrated from the ground up, which might be simpler in that you only touch what you actually need, but you also have to do it every single time, and it's different for each project/corp - with the vast majority missing out on the more complex bits that don't fit into an nginx config and some shell scripts.
I'd take a stack of yaml over a stack of bespoke aws managed service scripts any day.
> Yes, the standard approach is more complex. But it is better _in practice_ to have a single approach, rather than many individually-simpler, but in-aggregate-more-complex approaches.
There still remains such a strong reflexive "you don't need Kubernetes for that" response that shows up.
But we really are seeing a lot more people who get the value of there being a standard way of doing things, of having common platform & expectations underneath. The cost of starting up may be significant, but it results in a more legible understandable result, and we can keep reusing that base of capabilities again and again and again.
Its such an exciting change. Open source has been around for a long time, but running software has been the wild West, until very very recently. We now have this awesome, scalable, extensible way to manage containers, volumes, load balancers, databases, and whatever else have we.
Love to see it. Folks aren't wrong about it having complexity, but so much of the complexity already existed, was just not clearly visible or well harnesses (different example is like how systemd makes it so easy to restrict services, give them temporary/unprivileged users). We are closer to a real shared understanding, to having so many particular decisions offloaded & so much more we can work together atop. Love it so much.
If you're already using docker-compose, the jump from there to Docker Swarm is arguably tiny in comparison to the jump from there to Kubernetes, while still bringing most of its value.
So yes, I will keep evaluating the tradeoffs every time and choose accordingly.
There’s another thing that no one who advocates for these systems wants to mention: The cost of maintenance. I’m ok with systemd as 98% is outsourced to the maintainers. But I’d be more comfortable if k8s was a more monolithic system a la BSD. At least linux have distros.
It makes absolutely no sense to base this decision on the number of users. We have some applications that don't even have 10 users but still use k8s.
Try to understand the point that was made in the original comment: Kubernetes is a way to actually make infrastructure simpler to understand for a team which maintains lots of different applications, as it can scale from a deployment with just one pod to hundreds of nodes and a silly microservices architecture.
The point is not that every application might need this scalability, no the point is that for a team needing to maintain lots of different applications, some internal, some for customers, some in private datacenters, some in the cloud, Kubernetes can be the single common denominator.
Hell, I'm a hardcode NixOS fan but for most services, I still prefer to run them in k8s just as it is more portable. Today I might be okay having some service sitting on some box, running via systemd. But tomorrow I might want to run this service highly available on some cluster. Using k8s that is simple, so why not do it from the start by just treating k8s as a slighly more powerful docker-compose.
k3s is essentially a single binary and installed, up and running with a single command.
With a handful of command line flags you can have a full cluster up and running with some master nodes and some agents and high availability through etcd and load balancing and network policies and ingress and distributed clouds via WireGuard/Tailscale… They also have “blessed” options (that you can generally install with a single command) for things like persistent storage, etc.
I would not want to build a k8s cluster from scratch. Setting up and maintaining a k3s cluster is pretty trivial.
Personally, to me, this sounds like madness.
Deciding you need to custom tailor and brew up a specific solution that exactly matches each use case begets endless complexity. Constantly negotiating a position & changing approaches begets endless complexity.
And the complexity needle moves over time, such that your initial assumptions often don't stand. Getting something running usually isn't too hard. But creeping demands of observability, monitoring, rotating certs, backups, and other concerns are going to start getting tacked on. If you only ever do one thing, and there aren't many of ya'll & never will be, and very few newcomers will ever join, maybe maybe maybe you find a win today by doing less, by hand picking your options & getting in there & cobbling your stack of tools together. Convincing yourself to start small, to avoid "complex" Kubernetes to do less, is in most cases though going to backfire even for a single piece of software, as situations evolve.
There is such a fear that Kubernetes is complex, but it just feels like so many are missing the point, are resisting the standards & practices that erulabs' spoke to.
But man, it is so not complex to prop up a big k3s node & push some containers at it. And anyone whose spent more than 1 hours with kube will be able to look around, see what's running on the node, see how everything works; you won't have to explain what you did, it'll be obvious. And they'll be able to keep going. And you'll be able to keep encorporating concerns & platform & new apps in. And those skills - mastering that complexity - will transfer to every other thing you want to do, will scale way up and way down.
(I do think systemd offers a good number of similar advantages of replicatability, consistency, paved-roadness, and is pretty solid an option, but involves more finding-your-practice than kube)
With k8s you can easily install a bunch of helm charts and get a bespoke platform that becomes a full time role for multiple people.
There are pros and cons against this approach but you if you're worried k8s is complex, just use the cloud native integrations.
And not a collection of strung-together controllers.
Very much depends on the point of view. It's great from an SRE point of view but not necessarily application/developers, who are being constrained to a platform's idiosyncracies and expressions of platform egos.
The individually simpler solutions are only complex from a high level, ivory tower, or middle management perspective, not from the perspective of people who have to use and manage the application itself.
In the last 6 months my job has been to get production ready vault instances on azure. There's a bunch of complex, unreliable and not very fun APIs here.
Much like AWS, there isn't really a StatefulSet ( PetSet ) abstraction. To you need to write a bunch of bespoke logic where you figure which IP addresses, names, IDs, and Disks you need to attach to a new VM.
Whilst iterating, the azure APIs are eventually consistent that cause all sort of niggly problems. Resources that are created don't get added to TF when there's a failure.
I create a new vault on a PR and it takes 20 minutes to deploy.
The problem took a couple of months and I created a bunch of code.
On k8s I can just deploy the helm chart in 2 minutes. The abstractions are cleaner, it's more reliable, and way more fun.
At the end of the azure project, the team agreed to do a AKS poc, where we gave the task to the junior of the team, and the entire thing was done in a week.
If you're not doing this type of work, maybe you don't need k8s. But if you're not doing ephemeral test environments, do you actually not see them as a positive, or is it an effort thing? Because it takes me no real effort.
Then, it mostly works, so management makes "economy of scales" and removes all screw drivers, since they need more skills anyway.
Now, you are only using nails.
Until someone else rediscovers screws and starts another hype cycle.
I set one of these up on a few cheap odroid-h4's, and have quite enjoyed having a fairly automated (though quite complex of course) setup, that has centralized logging, metrics, dashboards, backup, etc. by copying/adapting other people's setups.
Instead of weechat, I went with a nice web based irc client (the lounge) to replace my irccloud subscription. kubesearch makes it easy to find other people's configs to learn from (https://kubesearch.dev/hr/ghcr.io-bjw-s-helm-app-template-th...).
There's nothing so special about weechat that it wouldn't work and you can just exec into the container and attach to tmux.
Running tmux/screen inside a container is definitely cursed, but, it works surprisingly well.
And at that point, it may as well just be a normal Ubuntu server.
I would personally consider that a bit of an anti-pattern (I would always want to have software upgrades tied to the container) but that makes sense!
Basically 3 paths for:
- things with well-maintained upstream helm charts (checks the latest version of the helm chart and updates helmfile)
- things where I wrote a helm chart (checks for the latest container image and updates the values.yaml)
- things where there is no (reasonable) public docker image, or my own software, where I have full CI/CD to build and deploy a new container image.
Probably takes me 15 minutes a week to keep my two personal clusters fully up-to-date.
docker run -itd ubuntu:24.04
...
docker exec -it df36 /bin/bash
...
root@df365a3d2257:/# apt update
...
root@df365a3d2257:/# apt upgrade
...But the author only uses it to keep alive a couple of IRC connections (which don't send you history or anything on re-connects) and to automatically backup their "huge" chat logs (seriously, 5 GiB is not huge, and if it's text then it can be compressed down to about 2 GiB — unless it's already compressed?).
That's not really the problem — if, say, one of your nodes drops dead (or just drops off the network), the clients' connections also drop, and they all try to reconnect. That just happens and there is not much you can do to prepare for it except by having some idle capacity already available.
Unless you're talking about rolling out strategies for deployment updates, and to be fair I don't remember the controls for that being all that useful but that was 2 years ago, so perhaps things are better now.
If you have a static number of nodes and don't have to worry too much about things autoscaling, I don't see why it couldn't be really stable?
Check out how services, load balancers and the majority of CNI actually work then.
Kubernetes was designed for stateless connections and it shows in many places.
If you want it to do stateful connections you could use something like Agones which intentionally bypasses a huge amount of kubernetes to use it only as a scheduler essentially.
Applications that act "native" and don't need a lot of the extras...
Well, they arguably mostly use just the scheduler then :D
Maybe I’m doing it wrong?
But you do not need a "complex" CNI. Originally k8s pretty much worked with assumption you can route few subnets in good old static way to the cluster and that's it, and it works with that kind of approach still - each node gets a /24, there's a separate shared /24 (or more) for services, etc.
The complexities came from the fact that a lot of places that wanted to deploy kubernetes couldn't provide such a simple network infrastructure to hosts, then later what was a workaround got equipped with various extra bells&whistles
Second: they use a version of node ports that bypasses CNI, so you directly connect to the process living on the node. This means there’s no hiccups with CNI if another node (or pod) gets unscheduled that had nothing to do with your process.
In most cases, web services will be fine with the kinds of hiccups I’m talking about (even websockets); however UDP streams will definitely lose data - and raw TCP ones may fail depending on the implementation.
No, why do yours? :D
If you're using cluster autoscaling with very small (or perfectly sized) nodes, I could see it being more of an issue on a busy cluster.
But even then, I wouldn't set up a database to auto-scale. A new node could get created, but it doesn't mean the db pods will be moved to it. They'd ideally stay in the same location. And on a really busy cluster, I'd prefer a separate node pool for stateful apps.
Using something like Stackgres makes it relatively painless to run postgres in k8s too, it handles setting up replicas and can do automatic failover.
Though, ironically, there are no CLI clients for its relay protocol, only for desktop/web/android.
curl 'https://news.ycombinator.com/item?id=41332427' | grep -i just
Need tables or something
Yes.
> I mean seriously, do communities hang around there?
Yes. They've been around for years.
But I’ve had the same Linux box for over 15 years now (through various hardware changes, I’ve kept the same Ubuntu install since circa 2008, using do-release-upgrade for major updates and making sure to keep up with security updates) and I’m not sure it’s really worth it to optimize for easy recovery from a fresh install. I back up my homedir and etc and some other important data, and if I had to rebuild my OS I’m sure I’d do it far cleaner this time and wouldn’t want to fully do things the way I had them anyway.
Even as a hobby it just doesn’t seem worth it to move away from the “pets, not cattle” model of servers when there’s just one or two of them (technically my router is separate but it’s a very simple router so there’s not much to do there)
Personally, I find docker compose to be the sweet spot for reproducibility and ease of use. There's some bespoke setup, but I can go from backups and fresh hardware to running apps in a couple of hours.
I switched to k8s entirely because I wanted to learn to use it, and I don't think I'd recommend it for anything else.
I had a ton of stuff that was important to me running in VMs on a Proxmox server. I had backups, but if anything had happened to that it would still be a nightmare. Most of it had been there long enough that my technical knowledge and knowledge of how I had set it up was rusty to nonexistent. Which meant that I was afraid to touch it. Which made me get rustier… It was starting to be a constant source of stress and anxiety.
Now I’ve got single repository that has everything that isn’t “actual user data”. I can (and have tested) taking a couple machines, PXE booting them, and within 15 minutes with no interaction getting to a fresh cluster. Then redeploying all my stuff takes me a couple hours (mostly due to slow internet) with, again, no interaction besides hitting “go” on a bunch of things. And god help me if I forget how Kubernetes works because then I’m probably out of a job too.
I switched to running on four Lenovo ThinkCentre Tiny PCs (like $40 each I think) as opposed to a single large box to provide some redundancy as well. Makes things like hardware failures not as much of an issue, upgrades less risky, etc.
I’m not afraid to do things with it anymore. I never worry that a piece of hardware failing is going to suddenly and unexpectedly mean a lost weekend. It’s made it all fun again instead of stressful and like work.
I never tried the actual multi-site support, but for my own homelab use-case - I don't have enough servers in each region to make their own ceph cluster.
Each write needs to be propagated before the application is unblocked. When you have tens of milliseconds of site to site latency this means every IO request is going to have worse performance than spinning rust.
Object storage can work better in this regard because the IO chunks tend to be larger and therefore the round trip time is traversed less per bit written.
Another alternative is that you run your regions in active passive setup so there is only one active region at a time and when the active region is changed all in process state in the previous active region can be sync to the new active region. Writes only need to be synchronously written within the active region.
Honestly, if you can do that you can make the big bucks at FAANG.
I personally just run a single home NAS which backs up to S3. Very strong durability in that S3 is never going to lose my data at the same time my NAS goes bad, but not great availability in that I will need to rebuild the NAS and pull down files from S3. When it comes to my home tech strong durability is a must but availability, not much so.
Signed, someone happily not writing YAML for k8s despite using k8s extensively