Hacker News Reader

Comment by ecma | Hacker News Reader

This is super interesting but

   due to the per-CPU inode table
   structure, it is impossible to
   move a NOVA filesystem from one
   system to another if the two
   machines do not have the same
   number of CPUs.

seems like a dealbreaker. What use is a FS if it isn't portable? At least it sounds like they're very aware of this issue.

I'd be very interested to see what they end up doing to make this behave better prior to upstream consideration. I wonder if a linked list journal of inode table changes (with space drawn from per-CPU freelists) would be safe/fast enough. That could be periodically coalesced and remapped to the NOVA device's non-CPU aware inode table.

View on HN

It's not so grim.

"The origin of the CPU-count dependence is that NOVA divides PMEM into per-CPU allocation regions. We use the current CPU ID as a hint about which region to use and avoid contention on the locks that protect it. So moving from a smaller number of CPUs to a larger number of CPUs just means more contention for the locks. Moving from a larger number to a smaller number is no problem at all. So, our current plan is to set the CPU count very high (like 256) when the file system is created." - comment under the post by one of the designers of the fs

Just wait 10 years until we all have 1000 core CPUS...

I don't see it as such a big problem. Most filesystem images aren't intended to be ported directly between machines in general. As long as they have a tool to read the filesystem on a different configuration, a filesystem with this restriction could still be very useful for generic bulk storage on a single node.

Agreed. They might fix it at some point, but if there's a performance advantage to making the filesystem non-portable between machines with different numbers of CPUs, many people might happily take that performance advantage, at the cost of having to do file-level backups/transfers rather than image-level.

What if you upgrade the machine with a new CPU that has more cores?

Disable some cores in the bios.

That sort of defeats the point of an upgrade.

So going from 2GHz to 3, or getting 50÷ IPC from an updated chip isn't worth it?

Archive and restore or transfer to new media.

A comment from one of the devs says there should be no impact as long as the number of CPU's stays below the number of locks the filesystem was created with, which will be very high by default. If it goes above, there is a possibility of more lock contention but it will still work.

Changing the number of cores given to a VM is not so uncommon.

CPUs with hyperthreading typically allow you to disable it in the BIOS. If you're running a new, beta-ish CPU microarchitecture (such as Ryzen/Threadripper) you might wish to disable HT as part of troubleshooting a CPU freeze/hang.

Changing a single CPU-related flag in the BIOS should under no circumstances render a filesystem unmountable.

> Most filesystem images aren't intended to be ported directly between machines in general

You don't have external hard drives or flash drives...?

This file system only works on nvdimms, which are typically in RAM sockets. It won't work on normal block devices.

Er, the comment was, "Most filesystem images aren't intended to be ported directly between machines in general."

The entire complaint itself is the fact that this file system isn't able to do what people expect from most file system images (read: pretty much every one, except this one). Citing the fact that this particular one fails that pattern isn't a justification, it's circular logic.

> Most filesystem images aren't intended to be ported directly between machines in general.

Wouldn't a Server hardware cluster potentially prove problematic?

This feels like a file system designed for mobile devices, where moving the storage volume would not be a relevant concern in most cases. Based on the design, it should be more responsive, use less CPU and thus less battery.

Simple answer may be treating it the way hash partioning treats shards, i.e. overallocate to start and map the over allocation to the physical layer.

From the comments, it appears that that's what they're doing:

> The origin of the CPU-count dependence is that NOVA divides PMEM into per-CPU allocation regions. We use the current CPU ID as a hint about which region to use and avoid contention on the locks that protect it.

> So moving from a smaller number of CPUs to a larger number of CPUs just means more contention for the locks. Moving from a larger number to a smaller number is no problem at all. So, our current plan is to set the CPU count very high (like 256) when the file system is created.

It seems like the developers could relatively easily add a slow compatibility mode that allows accessing filesystems from systems with different numbers of CPUs. It would just impose additional locking accessing all of the previous per-CPU datastructures.

Yeah this would be a deal-breaker. I can imagine the devs will be working on a way to "convert" one to the other, but I can also imagine that comes way down the priority list, so I am not disturbed by this fact, at this time.

This is required, I assume (after 5 min scan so can be wrong), that in order to speed things up, information about the CPUs are embedded in the filesystem.

So is this not a classic "hardcode for speed" vs "generic but slow" trade off?

Agreed, something they should have a plan for but not too worrying to see it with a few caveats at this stage.

I actually wrote a comment in response to loeg elsewhere in the thread and lost it by refreshing (and then lost interest, hah!) on the subject of conversion. I think there are some interesting problems around introducing a new FS and having to build "migrations" into it from the start. Migration feels like an expensive (in time/effort) word and I'd be concerned about issues coming up when you need to rewrite chunks of your NOVA FS because the per-CPU region has to become larger and everything else needs to shift along. It wouldn't need to be a full rewrite since only some data would need to be moved but it still seems clunky.

As mentioned in another comment, it seems like the solution being put forward is to oversubscribed the per-CPU data region for 256 CPUs but I wonder how long until that turns out to be a bad assumption. Who knows, maybe it doesn't matter. Maybe it's just out of scope for NOVA. One FS doesn't need to rule the world after all!

Reply on news.ycombinator.com