Show HN: Uberblok – an interactive museum of ZFS internals | Hacker News Reader

Show HN: Uberblok – an interactive museum of ZFS internals

uberblok.storagebits.workers.dev

126 pointsstoragebits39 comments

I've worked for a long time in the storage field during my career and ZFS is the piece of engineering I admire most in that space. I wanted a way to pay homage to it and to explore and better understand its internals. So I built Uberblok: a museum of ZFS. A deep dive into copy-on-write, txgs, vdevs, RAID-Z, snapshots, the ARC, metaslabs, the ZIL, send/receive with explanations and animated diagrams.

I'm sure people here know these internals better than I do, so corrections are very welcome.

I built it with my agent swarm organization, Algotitan.

I hope you'll enjoy the deep dive into Uberblok as much as I do.

Museum: https://uberblok.storagebits.workers.dev Repo: https://github.com/algotitanhq/uberblok

Open articleView on HN

I absolutely love ZFS. I use it on FreeBSD for all disks on the whole system, including the ones that FreeBSD itself is installed on, and I use it with some disks on Linux and with most of my external disks with macOS.

In an alternative universe, Mac OS X would have gotten ZFS shipped with the base install back in 2007, and presumably we’d then still have that to this day. In our universe that ended up not happening even though it was supposed to :/ https://mjtsai.com/blog/2025/04/28/revisiting-zfs-for-mac/

I really would have loved to have the whole of macOS running on ZFS. But at least using ZFS from homebrew on macOS to have ZFS on external disks is the next best thing after that, and which we do have.

I forgot that Apple story, a great "what ifs". It's not in the history wing yet, thanks for the link, it might deserve its own panel!

I began learning ZFS (TrueNAS and the macOS-plugin) just two weeks ago. It is absolutely incredible technology.

>I use it with ... most of my external disks with macOS

How do you personally deal with the terrible macOS default SAMBA client (if using SMB shares)? I can get saturated single-file transfers, but even deleting 100+ small files takes a longgggggg time. It is my understanding the the macOS only allows single-threaded SMB access.

I am considering using ExpanDrive (for NAS access only); openZFS is perfect for DAS (direct disk access), albeit not ideal (since the ARC takes away from system memory); is Thunderbolt linkage still impossible in TrueNAS?

How do you connect your mac fastest to TrueNAS (not top speed, actual IO rate)?

>> I use it with ... most of my external disks with macOS

> How do you personally deal with the terrible macOS default SAMBA client

I almost never use SMB. The external disks in question are drives that I connect via USB3 or USB4/Thunderbolt to the MacBook Pro, including a couple 5-bay multi-drive DAS enclosures that have USB4/Thunderbolt.

Wait until you find out that zrepl[1] exists, what it can do and that it can even be signaled ;-)

1: https://github.com/zrepl/zrepl

Currently zrepl is way above my pay-grade...

but if anybody else wants to read about this (the tagline "helps you configures zfs backups" – wasn't helpful to me) technology, here is the zrepl quickstart guide:

<https://zrepl.github.io/quickstart.html#quickstart-toc>

----

From zrepl guide, example use-cases:

•) Continuous Backup of a Server

•) Local Snapshots + Offline Backup to an External Disk

•) Fan-out replication

Just use an LLM, describe what you need and you get a commented config that does what you need.

I use it for:

- job 1: every 15 minute autosnapshot and cleanup

- job 2: push encrypted snapshots to an external harddrive via cronjob and signal

- job3 (another server): pull snapshots from the main server via ssh for ransomware resistant backups (mainserver cannot access backupserver at all)

Genuinely cool, but the LLM-speak started to wear on me.

An editing pass to make the prose less "punchy" would be great.

That's what that is! I was doing well until I got lost at Map of the Visit. I forgot I was pretending to be in a museum.

I'd love ZFS a lot more if it weren't for the ARC. Using ARC means that ZFS caches stuff outside the kernel page cache. That in turn means that sendfile is no longer zero-copy when data originates on ZFS, making it a non-starter for content drives for our CDN workload.

Yes, but this is a Linux/licensing problem. Use one of the BSDs and the ARC doesn't fight with the system cache.

Building a museum when you self admit you don't understand the internals that well... seems like a conflicting goal? If you're showing off to educate then how can you do that if you don't understand it?

We know the answer, obviously. I can't trust how much of this was something an AI said vs something you actually did seriously look over. At the very least, all of the text on this should've been in your own words, I think. Otherwise this is just show and tell of some prompts.

I wish I could enjoy tech demos anymore, but now they feel so hollow. Which is a real loss from the age of LLMs. I used to enjoy reading the dolphin EMU progress report for years. I wonder what they're up to.

Almost all of this was written by an LLM. No human writes like this.

As a mere operator, I've been fascinated by ZFS ever since I first heard of it 15+ years ago. It seems so elegant, like it really is the last generalist filesystem one would ever need. Performant, reliable, fundamentals easy to comprehend. I've never tried managing LVM configs because zpools and raidz are just so damned easy. Native encryption. Datasets can be as simple or as complicated as you need them to be. send/recv dataset syncing at the block level vs having to decrypt and sync files individually.

A lot of this goes over my head as someone who isn't familiar with filesystem internals, but I appreciate the effort and the homage to a great tool.

> Native encryption.

I think maybe that's the one feature that's iffy? Theoretically great but bad history of bugs. (Unfortunately)

I've been using encrypted datasets for years without problems. I don't used `send` and `receive` for backups though.

So many really brilliant people keep saying this, so I don't doubt that it's true. Can you explain to an ordinary user why ZFS isn't everywhere? It seems to be a marvel, yet ext4 lives on.

Everyone has mentioned the license, which is the reason ZFS isn't everywhere. I just wanted to add: Don't let the license scare you away. The CDDL is actually a nice compromise between copyleft (GPL) and copyfree (BSD). The CDDL is essentially the same as GPL except the GPL applies to everything it touches (include a single GPL *.c file in your project and now your entire project is subject to the GPL). Whereas the CDDL only applies to the CDDL-licensed files themselves. So you can include ZFS in your project without making your entire project subject to the CDDL.

The issue is, the GPL wants to spread to everything it touches (including the CDDL-licensed ZFS files), which would make the ZFS files subject to both the GPL and the CDDL. But the CDDL and the GPL are incompatible, which is why Linux can't include ZFS whereas more permissively licensed projects like the BSDs can.

So it's an unfortunate situation, but ZFS' license is not "bad". It's just the GPL and the CDDL are incompatible.

Because ZFS is complicated to use and isn’t necessary for most workloads.

The licensing issues don’t help. Consider that most Linux machines are in commercial settings (Google runs more of them than all individuals combined) and they don’t need Oracle on their backs.

> Because ZFS is complicated to use

There is nothing complicated about having zfs be your filesystem. You can use its extra features like snapshots, send/receive, etc, but only if you wanted to. Otherwise it's just a filesystem, there is nothing to use as such, so nothing complicated.

> and isn’t necessary for most workloads.

Not sure what that means. What workload wouldn't benefit from a reliable filesystem?

The fact that ZFS isn’t massively popular after close to 30 years, and after nearly 20 years of availability via OSS, suggests that the benefits aren’t really that meaningful to most users.

Others have answered this: licensing

Linux is the most common platform and many Linux distributions have an issue with license compatibility (or perceived one), which hinders adoption.

I agree partially. In the age of cheap backups and cloud storage, having super-high-integrity filesystems with easy RAID is more of a niche, so the need for ZFS on desktop and app servers is low. ZFS is fine as a general purpose filesystem, too.

In the end, I think it's momentum and licensing. AFAICT btrfs only exists because ZFS can't be shipped in Linux kernel. Defaults are insanely powerful, and ext4/xfs is good enough for a lot of use cases. If ZFS were BSD or GPL licensed, I think every Linux distro would ship with it as an option, if not a default.

I would say (in no particular order) that the CDDL license is not innocent, ext4 is good enough for many usages, cost ? ZFS is RAM hungry ... but ZFS is well present where it means most (NASes , Proxmox , backups to tell some)

Primarily licensing, IMO. Its license is incompatible with inclusion in the Linux kernel. Ubuntu ships with native support for it anyway; I guess they're not worried about legal issues. It's completely functional in Linux, you just have to take an extra step to install it, which is enough to keep it from mass non-techie adoption.

Totally worth it for NAS. I think there are more systems that utilize its advanced dataset features in the BSD world.

Thanks, that's exactly the reader I'd like to reach, and you're not the first to say it goes over your head. Honestly, I'm still trying to figure out who it doesn't go over the head of. Matt Ahrens, maybe? ^^ Next is a "start here" route: one plain question per room, answered before any code. Copy-on-write and snapshots are probably the best rooms to start with today.

One small suggestion: Make the code open to the right instead of a new window, that makes it easy to keep context without distraction.

Good idea, thanks! A side panel is the way to go.

Shipped, thanks to you: the copy-on-write room now opens its sources in a side panel. Your comment went from here to production in about four hours. Others rooms to come. The rendering is far better!

Sources in side panel is shipped on every page now, thanks again for the feedback! it adds clear visibility

ZFS is useless. No bit rot protection at all. ZFS just informs you about broken files (like FAT32 3 decades ago) but can't fix them... There should be some parity data added or some Salomon Reed forward correction of data... Waiting for such a feature for a decade now

If you can't be bothered to copy edit the gratingly AI prose, I can't be bothered to read it.

> I built

You keep using those words. I don't think it means what you think it means.

Fair hit. "Directed" is closer: agents wrote the code and most of the text, I chose the subject, steered and reviewed.

Nobody wants to land a "hit". They were hit. The hit was being shown something as though it were valuable, only to discover it is not.

You were a manager, in other words.

And yet, if you didn't do all that steering, etc... this wouldn't exist. So let them "be" little sore ppl.

ps. What I noticed is that those who created lots of projects before they'll be doing lots of projects after LLMs. Those who used to whine before, they'll keep whining after...

Inconceivable!

Nice work on Show. I'd be curious how it behaves once the state is stale.

Reply on news.ycombinator.com