New tricks for XFS: support for subvolumes
lwn.net
lwn.net
The same applies to ZFS (it's not possible to shrink a ZFS pool), which is why I'm currently using btrfs (with all its pain points) on my machines.
Possibly... how well does that work in practice? What happens when LVM runs out of space due to some runaway disk-filling process, is the filesystem ready to handle out-of-space errors coming from LVM in all situations?
Well, so is ZFS, but in some situations, like choosing what filesystem to use on a rented dedicated server, any choice that could lead to a situation requiring physical hardware configuration changes is a non-starter.
ZFS licensing is an issue too, as it means that you can't just boot into any ol' Linux live CD (or remotely boot into a rescue environment) to fix the system or salvage the data on it.
For backing up to another file system? How so? Is there something like "zfs send" or even "xfsdump"?
Honest question - I haven't look that closely at btrfs.
Sure you can. I've done just this with 3 different live CDs: ArchLinux, FreeBSD and OpenSolaris. I'm fairly sure I've also used ZFS on the Ubuntu Desktop live CD as well but that was just for playing rather than rescusing a degraded system.
Ubuntu desktop live CD doesn't contain zfs, you have to install it from apt.
However, if you have a ZFS system, I see no problem with having an USB stick with minimal installation of your distro of choice, together with ZFS support. I'm glad I did have it since the ZFS install.
> FreeBSD and OpenSolaris probably aren't very useful, when trying to rescue a Linux system. Especially if you need to chroot and run things from there. (My need so far was to rescue non-booting system, because the zfs package upgrade went wrong and didn't update spl too. Re-running dracut would be a somewhat problematic from these two systems).
I do see your point but it really depends on the problem as not all recoveries require chroot / package management access. I've rescued Solaris (not OpenSolaris) with an OpenSuse live CD back when a cavalier opp chmodded /etc. I've rescued OpenSolaris with a FreeBSD CD back when a faulty RAID controller borked the file system. As for ArchLinux ISOs, I've used them to rescue more systems than I can count. But as you said, some problems do just require booting an instances of the host OS via some means.
> Ubuntu desktop live CD doesn't contain zfs, you have to install it from apt.
It took me all of about 10 minutes to bake the ZFS driver into the ISO. It's not hard compared to the other technical challenges you've discussed. Though if that's too much effort then I think you can also just apt it from the Live CD and manually modprobe it into the running kernel.
> However, if you have a ZFS system, I see no problem with having an USB stick with minimal installation of your distro of choice, together with ZFS support. I'm glad I did have it since the ZFS install.
Indeed. My preferred method is having rescue disks available over PXE booting. Before then I was forever hunting down my recovery disks or spare USB keys / CD-Rs. Not to mention the pain involved if the system I was trying to recover was my main workstation (ie the hardware I'd normally use to download and burn CDs on).
Sure, there are very few problems that cannot be solved by throwing some time and sweat on them. However, when I do need to solve something, I prefer to not be sidetracked by sub-problems. Smooth sailing and all that.
It's much simpler to pull an usb key from the drawer or PXE boot, as you mentioned, and go on on solving the damaged system, than to start downloading and preparing a live distro somewhere.
Compared to grabbing standard media you have somewhere, it will take at least 15 minutes extra.
Basics of system administration does not mean, that you are wasting your time, especially on something you can be without.
You don't need a persistent root. I'd already addressed that point. Just run modprobe and you're done.
>Compared to grabbing standard media you have somewhere, it will take at least 15 minutes extra.
Bullshit. I've done exactly what I described and it did not take me 15 minutes. Furthermore all you're doing is pre-emptively pushing the work to before your outage which you could do the same with the ISO (if you really wanted to compare apples with apples).
> Basics of system administration does not mean, that you are wasting your time, especially on something you can be without.
The whole point of this tangent was about when one needs an Live CD. Not about whether creating a live CD is worthwhile when you already have a USB key. That new argument you've invented is stupid because the answer is quite clearly "use the USB key if that's already in your draw." But what happens if you have a ZFS volume on a system and you don't already have a recovery media? (ie the original question) Well in that case you can use any of the methods I described. Or, of course, you can create a USB key too. But that will take just as long as the methods I described anyway (you still have to download the OS image, ZFS drivers and write them all to your storage medium. Thus all you're really doing is swapping out one chunk of plastic with another chunk of plastic).
That assumes too much. For example, that you have a network connection while booted from the live media. You may not have one; then you cannot run apt/yum and you need persistent media that you prepared somewhere else. (Happened to me).
> Bullshit.
Surely. Or you have extra speedy USB keys. Just installing minimal distro on USB takes a better chunk of that time.
> The whole point of this tangent was about when one needs an Live CD.
When you are doing something non-standard - and installing ZFS on Linux is pretty nonstandard - you know in advance that the normal live media won't work. It's prudent to have something prepared, if/when SHTF event occurs.
Specifically with regards to filesystem, when you are installing with non-distro-provided-fs root, you need to make it anyway, just to install it in the first place. So instead of throwing it away, just label it and put in into the drawer. (When you are not installing on non-distro-fs root, you don't need support for that fs in live media at all, the standard one will do for making the system boot).
You've been assuming a crap load of stuff as well when it suits your argument. Like having a pre-prepared USB key to begin with.
> For example, that you have a network connection while booted from the live media. You may not have one; then you cannot run apt/yum and you need persistent media that you prepared somewhere else. (Happened to me).
Indeed. You might also not have a CD drive on the host (happened to me), or any blank CD-Rs, or a CD burner on your workstation. Or the internet connection might not work on your workstation either. But then most of those arguments can be made for creating a USB key as well so your point is moot. In fact my latest workstation (Macbook Pro) only has USB-C so I couldn't use my USB keys when I went to install Linux on that.
My point is, if you're looking for ways to nitpick, there are plenty for your examples as well. In fact there will be a thousand different exceptions for any solution you could dream up. Thus is the nature of working in IT.
> Just installing minimal distro on USB takes a better chunk of that time.
Arguably yes but that also takes longer and your original point was about getting stuff done as quickly as possible. So you're now contradicting yourself.
> When you are doing something non-standard - and installing ZFS on Linux is pretty nonstandard - you know in advance that the normal live media won't work.
Except the whole point of this tangent is me demonstrating where it does work.
> It's prudent to have something prepared, if/when SHTF event occurs.
Now you're arguing a different point to the point I was discussing. I'm not going to disagree with you there (since I've already discussed I run a PXE server for situations like these) but that wasn't the topic we were discussing.
I seriously just think you're now just arguing for the sake of winning an internet argument. I'm not going to argue with you that a CD is better than USB because it's pretty obvious that isn't the case. But that wasn't the point I was discussing. So for the benefit of my own sanity can we please get back onto topic: you can use live CDs to repair a degraded system running ZFS. Sure there will be occasions when you cannot; but that's the case when doing anything in IT (and thus why use sysadmins get to command such a good wage). But generally you can. And I literally have. Many times in fact. So enough with the dumb "death by a thousand paper cuts" and goal post moving arguments please.
You are still conveniently ignoring what I said: if you want to install system with ZFS root, you have to make it. That's also the reason why I have it. I just didn't throw it away after the installation.
> Except the whole point of this tangent is me demonstrating where it does work.
Yes, if everything is aligned right, it can work.
> I seriously just think you're now just arguing for the sake of winning an internet argument.
You are free to think whatever you want.
> you can use live CDs to repair a degraded system running ZFS.
Yes, under certain conditions. How they apply in your environment is up to you to assess.
> Sure there will be occasions when you cannot; but that's the case when doing anything in IT (and thus why use sysadmins get to command such a good wage). But generally you can. And I literally have. Many times in fact. So enough with the dumb "death by a thousand paper cuts" and goal post moving arguments please.
It's not goal post moving, it's what happens. Having a livecd that supports your configuration is advantageous to not having it. Being able to download a ready-mady one is advantageous to having to make it. Etc.
So when I can choose between freebsd or opensolaris iso and native system that fully support whatever I need (that was the original issue, remember?), of course I will choose the latter, or having the latter available is preferred.
I'm not ignoring it; I've repeatedly addressed it and pointed out how it's not true (the Ubuntu Desktop example). Want a few more examples? When I installed ArchLinux with a ZFS root I didn't use a custom ISO (read their ZFS wiki if you don't believe me). I also didn't create a custom Ubuntu Server ISO when I installed that with a ZFS root. Both were installed from CD - the vanilla CD available on their respective websites.
Also, even if you did install from a USB key; what's to say you don't then lose said key afterwards? I'm forever am losing them.
The point is whichever argument you're going to make will be full of more exceptions than you can count. So nitpicking one over the other, like you are, is an utterly pointless exercise and a distraction from the original point I was making.
> So when I can choose between freebsd or opensolaris iso and native system that fully support whatever I need (that was the original issue, remember?)
No that wasn't the original issue. The original issue was whether there are an live CDs that can be used to rescue a degraded ZFS system - which I've demonstrated there are.
However I do agree with you that running ZFS on Linux is a little pointless when FreeBSD and the OpenSolaris forks are all solid platforms and have unencumbered native ZFS support. Though installing a ZFS root on FreeBSD was just as painful as doing so on ArchLinux (at least that was the case a few versions ago - things might have improved since but thankfully FreeBSD never really needs rebuilds so I've not had revisit that particular pain point)
The situation has been quite reliable for some time now (single disk, raid{0,1,10}), moreover the feature set of btrfs is really wide (on par with zfs), with a very high flexibility: you can mix and match disks of different capacities, shrink and expand pools, change their redundance level through filtering, everything can be done online...
If it works for you, fine. Also it appears to get better - I won't touch it anymore if I can avoid it.
Honest question - my home box runs a Btrfs mirror on openSUSE.
That being said, unless I really need those specific features, I go for ext4 whenever possible, as it has to be the most battle tested one, at least when it comes to *nix. It also seems that fsck.ext4 has almost magical powers sometimes, but that shouldn't stop you from making backups obviously.
Related: http://events.linuxfoundation.org/sites/events/files/slides/...
It's easier to just backup and refill with a single set of reads and single set of writes than it is to suffer the huge overhead of all that random metadata updating and seeking.
And ZFS can be installed on any Linux distribution.
Yeah, I spent one afternoon migrating a CentOS+btrfs machine to XFS.
Btrfs is being obsoleted by RHEL only, because they have XFS expertise, but not Btrfs. No need to split limited resources. Other distributions will continue with development. SuSE doesn't intend to stop.
Looking at one machine available:
* / has 2.08 compression ratio,
* /var/cache 2.15
* /var/tmp 1.0
* /home 1.02
* /srv 1.10 (few web apps, pgsql instance for these web apps, svn repo, samba shares, local prometheus store).
If your server uses spinning rust, you can also increase the I/O by using compression. You are trading the CPU time for I/O bandwidth. Depending on your workload, it may be a sensible trade-off.
But still wondering: does the amount of this data really warrant compression? I mean the smallest sensible size of an SSD is around 100GB and it has lots of performance.
The compression was just something nice to have. Lz4 in filesystem is basically for free.
My first test of PostgreSQL on ZFS was also quite instructive. lz4 again achieved respectable compression (>10× for my datasets) and improved throughput several fold (with no other tuning!).
I have to admit that recently I've only had the exact opposite use-case: Wanting to expand volumes.
Except for authoring install-media or images where you want to reduce the final file-size... What common use-cases are there for volume-reductions?
... I’ve happened to spend much of the last three weeks learning about XFS.
In the SGI case it is called "inst"
Something I wish there was a better solution for. Perhaps there is a filesystem out there that supports CoW in a way that fits this usecase (XFS perhaps even), but I haven't looked into it.
The disadvantage of using hardlinks is that you can't hardlink between users (user is a property of the file, not of the link to the file), and there's always the danger that a write takes place through one of the links. Imho, that should really be solved at the filesystem level using a CoW scheme.
I'd recommend zfs snapshots - or for something a bit different, maybe nilfs2:
To me it sounds like you want simple snapshots and backups without the redundancy at the storage level.
So why don't use a filesystem which supports that natively, like ZFS or Btrfs?
Anyway, theoretically[0]:
> As you might know btrfs treats subvolumes as filesystems and hence the number of snapshots is indeed limited: namely by the size of files. According to the btrfs wiki the maximum filesize that can be reached is 2^64 byte == 16 EiB
But it seems practically you're hitting the mud at ~100 snapshots[1] but be sure to read the reply to that mail as it will depend on the use case and it might turn out to be fine way beyond that.
[0]: https://unix.stackexchange.com/questions/140360/practical-li...
[1]: https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg...
*not the best data point I realise....
Hardlink is not CoW. Modify the file all your backup will change too.
edit: Local in this case means cloning inside one filesystem. Hardlinks cannot span filesystem boundaries of course.
You're probably not doing your ongoing work in the branch you are merging for release anyway, if you did you'd have solve any conflicts all over again when you switch clone.
Is this a really a precise description? I was sure that (for data) the actual copy only happens in case of a block with multiple references. If there was a single use of a block I expected it to be modified in place in most real-world filesystems. Summary from wikipedia seems to confirm that.
Am I missing something, or is that just unfortunate wording?
What they describe with the write of the data followed with write of the indexes as new elements seems more like a log filesystem.
I'll stick to ZFS :)