LXD is now under Canonical
linuxcontainers.org
linuxcontainers.org
Also, can confirm void is amazing on the desktop.
The one gotcha to be aware of here is that libvirtd as of now has two ways to be operated: Either run just the monolithic libvirtd service, or individual subsystems like virtstoraged. Trying to do both at the same time will cause issues.
Due to the limitations of OpenRC compared to systemd for interdependent and interoperating system services like this, I stuck with the monolthic libvirtd approach and it's been running fine so far.
Monolithic libvirtd (which is disabling the systemd libvirt) is the way to go.
I was also aiming for Xen initially. However, my host just crashed (can't recall the exact dump) when I tried running a Xen kernel - Alpine or not. I never confirmed if it's due to the Intel N5105 lacking the proper CPU extensions to support Xen or if I could hope to hassle the machine vendor for a firmware update to get it working.
Main benefits is setting up a new machine is much easier, and updates are easier because almost all the normal configuration is managed by nixos (in the nix language) as opposed to regular files floating around etc.
>The LXD section on the LinuxContainers community forum will slowly be sunset in favor of the Ubuntu Discourse forum run by Canonical
A lot of useful lxd information are in forum.Stephane and the team is super helpful there.
Also snap auto update kills my containers so I had to put on held.
They also had just fixed a long standing bug that makes rebooting lxd container sometimes fails due to snap messing up with zfs mounts.
So far it's very snappy! My Ubuntu 18.04 machine somehow "rotted" in ways that previous installs didn't -- everything became slow and janky, sorta like Windows
And I specifically installed 18.04 in 2021, to avoid Snaps
But now it looks like Debian will work great
That is, IIRC, you get (say) Firefox via Debs in Pop. Canonical wants to install it via Snap.
IMO Flatpak is a great option for a set of desktop software where packaging across distros may be problematic for some reason. I use the Firefox Flatpak on Fedora / RHEL because it doesn't disable the video codecs, for instance. The native package doesn't have some codecs enabled.
Canonical is (AIUI) pushing Snap for desktop and server software. And Canonical is the only source of Snaps, I believe? With Flatpak you get Flathub but AFAIK anybody could set up a repo of Flatpaks.
You can use them side-by-side on the same system. Install a dozen of each and take some measurements.
For those who don't get the reference: https://en.m.wikipedia.org/wiki/Snap,_Crackle_and_Pop
I kinda like the window tiling, but it's still GNOME and GNOME is still a pain.
As a cleaned-up Ubuntu, I like Zinc.
https://teejeetech.com/2022/05/07/zinc-22-04/
Ubuntu, with the smallest cleanest but richest desktop on Linux -- Xfce -- and neither Snap nor Flatpak. Instead, `deb-get` which finds and installs native DEB packages, configures the repos for you, manages updates and so on.
I have suspend/wake working well so far, USB-C charging + screen + all peripherals via 1 USB-C cable, all "media" buttons work. And this is still on 8 GB of ram (soon the 2x32 GB will arrive ;)), I haven't even heard the fans so far! I'm on Teams, camera works, nobody would even know I'm on Linux if it wasn't for my constant evangelizing!
I'm really impressed (yeah, I know, Linux people easily are when it comes to desktops, basically we're impressed if things work).
I did enable AppIndicator. Solaar, NextCloud and Tailscale all really require it to be considered functional. I don't understand why that is not a standard thing (it is on many distros though).
... Sweet?
I don't like GNOME. I don't like GNOME accessories; I hate CSD and hamburger menus and that big empty wasted top panel. I want more things vertical, while GNOME is moving its vertical workspace-switcher to horizontal.
The Cosmic tiling is good. That's an improvement. GNOME's window management sucks, and this is better.
I also really don't like systemd-boot and it broke one of my laptops severely.
So, I am happy that it is good for you, but it is not something I'd choose.
...Seat, sweat, chaste, caste, Leigh, eight, height, Put, nut, granite, and unite. ... [0]
And I do agree on the very high amount of pixels wasted in the top panel, expecially on an ultrawide monitor, something MacOS does better. On a laptop screen it's ok for me.
As for the panel... well, Dash-to-Panel tames it:
https://extensions.gnome.org/extension/1160/dash-to-panel/
... but it does not handle vertical mode well. E.g. Status icons should be in rows, not a single column.
OOTB the Gnome install was as good as can be. It's a nice way to play with Nix, batteries included.
Right now I need to deploy a server used for bioinformatics, and I need Conda... And that is a pain [0], so again I'm on the fence: Deploy Ubuntu or NixOS and persevere... I'm thinking Ubuntu, then perhaps later in the learning curve I go for Nix or perhaps just make a container to use on Nix.
https://github.com/pop-os/cosmic-epoch
When I can I'm creating servers with Debian too, personal ones and for customers. The only problem there is that Letsencrypt uses a snap to update the certificates, even on Debian [1]. Not all my servers need a web server but when they do I usually use ngnix. I'm investigating alternative update clients with no hurry. When that is solved, goodbye to Ubuntu.
[1] https://certbot.eff.org/instructions?ws=nginx&os=debianbuste...
> If you have any Certbot packages installed using an OS package manager like apt, dnf, or yum, you should remove them before installing the Certbot snap to ensure that when you run the command certbot the snap is used rather than the installation from your OS package manager.
Yes, it is more work, but since then I've been upgrading Debian and it just gets better!
Oh man, this reminds me of two years ago. During Covid lockdowns, I figured I'd try Ubuntu, see what's up with gnome. This was 20.04. I was pretty meh, but it was okay. Then I found out about Regolith [0], which is a "spin" or whatever it's called, that comes with i3 (like Kubuntu has KDE instead of Gnome).
I don't know exactly why, and I was too lazy to dig, but that thing was one of the most sluggish experiences I've ever had. At the time, I'd daily drive an i5-6500 with integrated graphics, and my regular Arch with i3 and light Picom effects (blurring for notifications and dimming inactive windows) worked great on the integrated GPU. Somehow, the Ubuntu version was worse on a newer i5-8500.
On Snap... Even if you like Snaps packages, I can't imagine that you'd feel like it's a good choice for LXD. For a browser or email client, sure it's a solution, I suppose, but for thing like LXD it just adds an entire level over complexity that doesn't help. I'd go so far at to say that Snap packages makes LXD unusable, it's simply to complex, to messy. Canonical also have Microk8s (I think) which is packaged as a Snap, rendering it completely useless. The mental overhead required to make it function completely eliminates the point.
It also completely ignores highly regulated businesses, because Snap does not care and will just roll out an update whenever it feel like it.
I don't particularly like Snap, but I can see the use on desktops. For the server side... I don't know anyone who'd thinks it's neat.
https://snapcraft.io/docs/keeping-snaps-up-to-date#heading--...
If there's an option, live with it and change it.
I can't see what's so destructive about snaps. It's probably another echo chamber when people find a common target to yell at.
The packaging is very limiting, and I don't want to have to fight against the system to get a version of software installed that works.
I personally used to run MicroK8s (snap based). I finally gave up about 6 months ago. Were snaps the root cause? Eh... no, the root cause was dqlite synchronization problems that I was tired of dealing with, but snaps made the whole process so much worse. They're hard to automate, they're hard to interact with using normal tooling. They just aren't part of the system in the same way, and while I understand there's value there (hell - I'm running containers and microk8s for a reason) there's also pain. K8s was an okish trade on that front: I felt a lot of pain and gained a lot of value.
Snaps seem to be falling into the worst possible spot: I'm feeling a lot of pain and the product is usually not any better at all (and sometimes much worse).
HA Microk8s with snaps and nvidia-container-toolkit (The GPU addon) just isn't all that stable (I would consistently get hard lockups on some hardware). That same hardware running Arch and k3s is chugging away beautifully.
So maybe it is an echo chamber - but the echoes I keep hearing are system admins saying this is making life harder and they don't understand the upside, and Canonical pushing forward anyways.
---
So maybe there's light at the end of the tunnel somewhere for snaps. Maybe the grand vision will eventually come and folks will be fine with it. But my suspicion is that Ubuntu is being replaced at most orgs before then.
Steam is throwing weight behind Arch (LOTS of new linux users coming in here). Alpine is a strong contender in containers. Nix is interesting.
Both Flatpak and AppImage seem to achieve the "easy user installs" part without all the pain of snaps, and they're neutral on whether I use them or not.
Basically - if snaps were so good (even if they're only good in some cases) I would expect a conversation about snaps similar to that of SystemD - Strong arguments on both sides, real advocacy from the community for the product, even if it's painful to adopt, or doesn't meet all user needs.
Instead snaps are just "meh" all around. There's a steady stream of folks leaving, and no real advocates outside of Canonical. At most - you have neutral folks like yourself who just don't mind snaps.
To me - that's a bad sign. I bailed. It wasn't worth betting my personal projects on. I've been pretty happy with the decision too.
The implementation is shit.
Things take 10 times as long to start.
There are annoying 'Turn off firefox so snap can update it' messages. Nothing happens when you stop firefox, you'll just get the same message a few hours later.
Shutdown is going to take ages for some reason because it's waiting for snap to do something.
I don't give two shits about whatever snap is trying to do, but having to wait 10 minutes for my workstation to be useful whenever I want to reboot is not what I'm about.
SnapD is always the top process. I actually started a migration to Archlinux, which has LXD as native packages. Insane. I know.
snap refresh --hold
was made available my team went through the pains of snap workarounds [1]
The LXD team was very proactive in the support side but the truth is that Snap is a pain and without --hold it would be possible for production servers to break out of a sudden due to uncontrolled updates (who did every think that this was a good idea?).
LXD itself is a very good project because it uniformizes the use of containers and VMs with a friendly CLI - the syntax is more intuitive than the one from Docker, by the way. We recently managed to combine LXD and Puppet to manage containers and VMs in a declarative way. I am a big fan of the LXD project.
Somebody else commented here that the latest Debian makes LXD available in the normal distribution packages - that is something worth checking.
[1] - https://discuss.linuxcontainers.org/t/repeatable-lxd-install...
I will say that I prefer running LXD on NixOS hosts where it isn't packaged as a snap. Hopefully canonical doesn't somehow break that.
Same, though it seems this acquisition is mainly about snaps, so we'll see.
I wish Nix would package Kata, a hybrid of containers and VMs. I would migrate to that from LXD.
It does have Firecracker and Ignite though, which I haven't tested but probably also suffice.
You could run podman or other OCI containers inside LXD. I use LXD to test multi node K8s (K3s) on my desktop system.
I have used both lxd and podman.
In addition, there are orchestrators which can run LXC containers (LXD is a management layer over LXC). Hashicorp Nomad is noteworthy.
Added later: K8s runs OCI containers. All OCI containers I have seen are application containers. I don't know if OCI specification supports system containers.
[1] https://ubuntu.com/blog/lxd-virtual-machines-an-overview
[2] https://developers.redhat.com/blog/2019/04/24/how-to-run-sys...
LXD is a management layer over LXC and Qemu(KVM?). LXC is all about system containers. The Qemu support is a recent addition [1]. LXD supported only LXC system containers until then.
> Containers you run with docker and podman are fully capable of running multiple processes
Yes. I have done this. But it was very unwieldy - probably because docker, podman etc weren't designed to run system containers.
[1] https://ubuntu.com/blog/lxd-virtual-machines-an-overview
Make sure to post the article on HN if it comes around, I'm sure it will be appreciated.
That is something I'm yet to try. I will post whatever I can on HN. Thanks!
But podman also supports systemd inside a container and along with macvlan networking you can pretty much build an "independent" container acting almost as a VM.
Would LXD provide any other advantages/differences to that?
The original idea behind LXD was to build packages in a system equivalent to your real system but without spamming your dpkg database and filesystem with all the required -dev files. The idea behind Docker (which podman is a better implementation of and extension to) was to be able to have three different versions of Apache ephemerally on your system to match the versions used by three different projects today.
Ultimately you can do both with either (and I generally use podman to build packages, as an example) but if you do that you aren't playing to the project's strengths.
Toolbox and distrobox are sort of middle grounds that do both of them well enough to be tolerable.
One advantage of having a system container is you can use a package manager to update your applications. With Podman, you have to replace the entire container, with LXD you can just "PATCH" the container. This results in much faster upgrade / lifecycle management.
You can read about this on my blog post. https://zacksiri.dev/posts/why-i-created-pakman
I've created a project called instellar https://instellar.app which uses LXD under the hood. I'm working towards open sourcing instellar. It basically automates continuous deployment pipeline and automatically manages your infrastructure. Basically you get a heroku in a box.
I see a lot of people complaining about snap here. In the initial days yes, snap's auto upgrade would bring down a cluster, however I believe they've resolved this issue. Currently from my experience things are really smooth between upgrades. Basically invisible to me.
Hope this change brings LXD forward, and while the team may "regret the decision" I do hope they continue to work on LXD.
It's rather hard to imagine LXD continuing as a going concern without him. Does this mean he regrets the decision, but is going to keep working on LXD, or... does this mean LXD is fucked?
-- concerned LXD user
It is either way. Docker / kubernetes is eating it up. There aren't as many LXD users as they'd hope.
Oh and he does work for Canonical.
Is LXD's purpose the same as Docker/Kubernetes? I use both LXD and Docker and to me they are tools that use the same technology (containerzation) but for different purposes. Docker is for stateless containers, used to containerize services. LXD is for stateful containers, used to containerize operating systems. LXD can also run VMs while Docker can't.
I think about LXD as a sort of Proxmox alternative (because you can manage LXC containers and VMs with it) instead of a Docker/Kubernetes alternative. In fact, I actually have migrated my systems from Proxmox to Ubuntu Server with Docker, for stateless applications, and LXD, for when I need to containerize an entire OS or VMs for my friends.
So commands like this work on both containers and VMs:
lxc start fedora-desktop
lxc mv pi-hole other-machine:pi-hole
It did occupy the same space including Proxmox.
Back then KVM, Proxmox, VMWare, OpenVZ, LXD, etc were all ways of virtualization.
Whilst it's not exactly the same purpose most companies have or have been shifting to docker and kubernetes and in the cloud vs running their own VM infrastructure.
> Docker is for stateless containers
Tell that to people that run databases with docker and kubernetes. It is a thing and even commercial products are built that way.
To be honest my original message wasn't clear enough: What I wanted to mean is that I like using Docker when I have a container image ready to run, "stateless" in the meaning that the image itself is unmodifiable and any changes to the image will be lost after a restart. (This doesn't include bind mounts, and stuff like that)
LXD is nice because it has a nice management layer. It is really easy to migrate an instance (container or VM) and its state from one physical machine to another on the LAN, even if you don't want to bother with setting up an actual cluster.
Don't get me wrong, Proxmox is a pretty good piece of software, but if your workload isn't tailored to VMs + you have private links between your clusters + you have shared storage, you end up adding way too much complexity to your stack even tho you aren't using the real useful features that Proxmox provides.
So if you are in the "I just want to run my services" crowd, an Ubuntu Server (or any other distro, really) running Docker on baremetal + LXD for anything that can't run on Docker is way simpler to manage. Especially because running Docker on Proxmox is not fun (too cumbersome to run it within a LXC container + ZFS, running a big fat VM with Docker defeats the point since you can't backup individual containers with Proxmox anymore, and running Docker on the hypervisor is a big no-no)
At the end of the day, nothing in Proxmox has a special magic sauce that makes it tick, and sometimes that complexity may be super cumbersome when you just want to run some dang Docker containers for your swifty new app. https://mrpowergamerbr.com/us/blog/2022-11-20-proxmox-isnt-f...
Therefore I don't think there is anything that can run on LXC/LXD but not on Docker (and vice versa); it's more a matter of preference, whether you want a long-lived persistent virtual system (LXC/LXD) or ephemeral single-application containers with optional persistence (Docker).
Still, nothing stops you from using LXC/LXD for ephemeral containers and Docker for long-lived systems, but you'd be using the non-optimal tool.
Also, from a development standpoint sometimes a long-lived container environment is easier. I run zigbee2mqtt inside of lxd because if I want to try a PR, it’s `git checkout … && npm ci` and not building a whole container each time.
For the home NAS and server case, I’ve been really happy with Ubuntu server with zfs + lxd + docker. And, a lxd VM for Home Assistant OS. Basically, the right tool for each job and no worries trying to force software into an environment their developers don’t expect.
But how do I use LXD on Debian without Snap?
> While the team behind Linux Containers regrets that decision and will be missing LXD as one of its projects, it does respect Canonical’s decision and is now in the process of moving the project over.
I'm curious how that works; if the owner isn't happy about transferring the project, why go along with it? What leverage or right does Canonical have?
I wish LXD team try to keep it open and stop entangling lxd with the rest of canonical tools.
We do not deploy lxd on scale using snap. We build it and have as part of our packer pipeline on rockylinux 8.
Completely unnecessary, 100% visual, and made everything worse.
I bailed from them shortly after that, even with the option to use other GUIs. As Linux desktop has matured, it hasnt been necessary to go back to Ubuntu.
I imagine most of Canonical's power comes from users who don't try new things, or corporations who are too big to change anything.
Which one was that?
No version of Ubuntu I've ever seen has ribbons. This is one of the reasons I like it.
Do you mean when they switched from Unity to GNOME?
I suppose if I'm tied to anything, its the Debian branch of distros.
I loved Unity. I still use it today.
With that said, LXD currently builds from source and can install on Debian without needing snap. Canonical's VM counterpart "multipass" while on github cannot (easily) be built from source code and requires snap to install.
I very much hope LXD does not go that route.
* Ubuntu powers the overwhelming majority of Azure workloads (both customer and internal)
* Nearly all Linux applications and tooling target Ubuntu first (or at the very least)
* Github, VSCode. The missing piece is the actual runtime operating system. Ubuntu.
* Canonical employs nearly all the remaining core enterprise open-source developers after IBM acquired Red Hat
Start the countdown to Microsoft Ubuntu.
What? You greatly overestimate Canonical. Buying it would be a rounding error in Microsoft's balance sheet.
Probably far more devs than they'd get trying to make their own MS-branded Debian from scratch. Even if a bunch of them left after a while, they'd have competent, experienced people working for them for that time, instead of trying to staff up from zero and having a lot time where there aren't enough people to really get much done, and during that time they can hire new people who aren't going to quit just because it's MS.
And a couple from Rust side, after the Mozilla layoffs.
And a couple of ISO C++ members.
And their own Azure Linux distribution went out of preview at BUILD 2023.
Under her leadership, the FTC has started doing the right thing (finally!) and saying "No, not happening!" to some of the proposed mergers in very recent times.
Microsoft created Clippy. They also have a large partnership with OpenAI.
Given these facts, it is in the interest of the free world for Microsoft not to have the ability to maximize paperclip production at this time.
If Biden (or Newsom or whoever) squeaks in, they'll be looking to appeal to the right for the midterms, and lightly smearing and firing Khan might actually help them. The WSJ, in particular, has declared her enemy no.1, and the Biden WH has almost seemed embarrassed about the only person in their administration that could get them unambiguously good press, because they don't actually support her.
Why didn't they make Khan their "Dr. Fauci"? Why didn't they push social media companies to amplify accounts that supported her work, or to suppress accounts that spread misinformation or negativity about it?
Might it happen? Sure.
But nothing seems to have suddenly increased the likelihood of this recently, so I'm not sure where the "very soon" is coming from.
Other than maybe you just personally realised they're not badly aligned.
I expect this to shift. Some years ago `FROM ubuntu:something` was a go-to for Dockerfiles when I wasn't feeling fancy with `FROM alpine` or `FROM busybox` (or `FROM scratch`). Today if I need a generic distro base, I'm consciously going with Debian-based images and Ubuntu is never an option anymore (it used to be "Debian, but better", it became "Debian, but worse").
And I strongly suspect I'm very much not alone. So give it a few years while people convert and upgrade stuff and this might be not exactly longer true.
I used to use Debian base image, but now switch to Ubuntu LTS base images almost exclusively due to them offering newer versions of most core libraries, and longer support duration.
Is that actually true? Debian has managed pretty consistent 2 year release cadence, same as Ubuntu LTS, so based on that alone they should on average have similar library freshness. Basically Debian releases on odd years, Ubuntu LTS on even years. I suppose Debian might have slightly longer freeze periods.
docker.io/debian:bullseye 123.4
docker.io/debian:bookworm 115.7
docker.io/debian:bullseye-slim 80.1
docker.io/ubuntu:22.04 76.6
docker.io/debian:bookworm-slim 74.2
docker.io/ubuntu:23.04 69.4
I'm sorry, but what? This is not even remotely close to being true. There's even a pretty good chance that Red Hat employs more "enterprise open source developers" now than in 2018.
Red Hat 2018 year result was $3B in revenue https://www.redhat.com/en/about/press-releases/red-hat-repor...
and Canonical's 2021 revenue was $175M https://techcrunch.com/2022/04/21/canonical-now-hopes-to-ipo...
I wonder how LXC will evolve and if another product will arise from this? It seems like the Linux Containers organization is staying separate with LXC.
And to be honest, Snaps had fair chance of succeeding; when they first were released flatpak was still pretty young. Docker has had its own issues, and podman (etc) is even younger yet. If they played their cards right Snap could have been Canonicals chance to shine, and I guess they know it and are still trying to salvage the effort.
Agree it would be vastly more popular and interesting if it didn't require snap.
In my case, because it's being forced and replacing applications that worked out of the box with applications with issues. Firefox, for example. After it became a snap, you'll see a message telling you to close Firefox so it could update itself (?) but simply closing it is not enough and, last time I checked, it looked like you had to run some command line to solve it. Also, due to its sandboxing, external hardware like smart card readers didn't work in the snap version, so I couldn't log into e-government websites anymore.
I ended up installing the tarball version...
Something like an IDE is a better example since its not much of an attack vector.
Because otherwise, the result is people pissed off with that change.
One of the problems I have with Snap is that it does auto-updates. So you can't control when updates are performed. Furthermore, you can only use Canonical's repository. So if you want to use a custom build of some sort, you can't - unless you pay big bucks to Canonical.
1. https://snapcraft.io/blog/hold-your-horses-i-mean-snaps-new-...
Contrast with something like systemd that also has its haters, but it is solidly written software so seems to be accepted by a lot more people.
Couldn't that description be applied to the existing Linux packaging ecosystem? It seems that "fragile and complicated" dependency graphs are exactly what Snap/Flatpak/AppImage are trying to fix.
There are two things that Snap and Flatpak are trying to 'fix':
1. The Unix security model, which relies on the user as a main boundary for security/policy. These tools are trying to add sandboxing that makes running untrusted software safer and accounts for the fact that being able to access everything in a user's home dir (and so on; this isn't just about the filesystem) is 'bad enough'.
2. The Linux packaging ecosystem (and whole userland, down to glibc's ABI policies) has never been intended to serve proprietary software developers/publishers/vendors who want to be able to throw their binaries over the wall or test only on an extremely limited range of library versions.
(AppImage only aims to 'fix' the latter, and poorly at that, since it doesn't even attempt to encapsulate huge parts of the runtime environment that vary from distro to distro.)
The only thing 'fragile and complicated' about Linux packaging itself is the practice of installing everything into a shared global namespace on the filesystem, which is already better addressed within the packaging infrastructure itself, as has been done on NixOS, GuixSD, GoboLinux, and Distri. (The latter two being more research projects than practical distros.) Those distros all prove that resource sharing and dynamic linking are actually red herrings when it comes to what is problematic about distributing software on Linux— you can keep both of those things and escape dependency hell forever (with or without containerization).