New NTFS Read-Write Driver from Paragon Merged to Linux Kernel | Hacker News Reader

New NTFS Read-Write Driver from Paragon Merged to Linux Kernel

git.kernel.org

176 pointsbeshrkayali56 comments

Does anyone know what confluence of events or motives made this land now? I'm grateful for their work--just curious why a commercial software company decided to make its main capability available for free.

Some context: https://arstechnica.com/gadgets/2021/08/paragon-is-working-t...

So just a guess; maybe after Samsung's exfat code was merged into mainline, license and support revenue from their own exfat driver took a hit. So they decided it's in their best interest to preemptively get their own ntfs driver in mainline thus they can get more support contracts this way?

PS: I noticed this time it's Tuxera who isn't happy with this development. They remembered they maintain ntfs-3g [1], and decided up the version number of ntfs-3g after 4 years [2]. So I guess the theory above makes at least a slight sense.

[1] https://lore.kernel.org/lkml/alpine.DEB.2.20.2109030047330.2...

[2] https://github.com/tuxera/ntfs-3g/wiki/NTFS-3G-Release-Histo...

There’s an official statement here:

https://www.paragon-software.com/home/ntfs3-driver-faq/

Reading between the lines, I suspect that the effort of maintaining an out-of-tree file system outweighed any money to be made from selling NTFS support to Linux users, especially when ntfs-3g is probably good enough for many use cases. They have a lot of other products, and it’s possible that some of them leverage this NTFS support as well.

Anything out of the kernel source tree is pain in the ass for many reasons, ranging from a nuisance to impossible to use due to tainted kernel nullifying OS vendor support guarantees.

This was my main question: What does this solve that ntfs-3g doesn't? I thought issues with NTFS on linux were a thing of the past.

Performance. FUSE filesystems are cool but not really native in the same way an implementation like this is.

And "performance" here isn't some single digit percentage niggling either, it's something like a 4x difference. Enough to genuinely be a pretty big deal for any usage beyond minimal and occasional "get some data off this fs once" type of usage. Of course really the answer is that Microsoft should just adopt Lundman's efforts and add native OpenZFS support to Windows but in the mean time... ;).

And ksmbd is in Linus' tree.

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

For readers who don't understand what that is or how that's related, ksmbd appears to be a modern kernel-side implementation of the SMB (Samba) protocol which is used for sharing files across machines. I believe that's the protocol some people use to interop with Windows Active Directory for example.

I’d love to know once this is merged and has journaling if there are advantages to use NTFS instead of ext4 or other file systems as your main Linux FS, versus interop with windows drives.

IIRC, NTFS doesn't support permissions.

config NTFS3_FS_POSIX_ACL

bool "NTFS POSIX Access Control Lists"

depends on NTFS3_FS

select FS_POSIX_ACL

help

POSIX Access Control Lists (ACLs) support additional access rights for users and groups beyond the standard owner/group/world scheme, and this option selects support for ACLs specifically for ntfs filesystems.

NOTE: this is linux only feature. Windows will ignore these ACLs.

I'm not sure, but iirc POSIX ACL and typical Unix permission (ala chmod 0755) are two different things.

Unix permissions are those you change via chmod while POSIX ACLs are those you view/change with getfacl/setfacl.

NTFS probably has that because Windows used to have (still has? dunno) a full POSIX subsystem / compatibility layer.

NTFS actually does support permissions, it's Windows that doesn't. NTFS also supports symbolic links and hard links.

Not contradicting just genuinely wondering, what do you mean by windows doesn't? I have different permissions setup for various users on different folders on ntfs volumes in regular windows 10 installs. Or do you mean another sort of permissions?

In this context they are talking about the POSIX permissions related to running Linux on the drive which NTFS supports but Windows does not use. Instead Windows uses another attribute for their security descriptor model which is also used by other non-file objects throughout the OS.

A lot of this stuff is actually legacy from when NT hosted POSIX and Windows subsystems simultaneously on the same kernel.

Sadly that stuff is long since dead but they have in part revived some of the ideas with their SQL Server for Linux port and WSL of course.

Not sure why I'm getting downvoted here. When POSIX subsystem of the NT kernel they added support for many POSIX style features. Namely POSIX ACL support, hard links and case-sensitivity.

I guess the reality of this is lost to history and folks like me that remember NT for what it was before it was relegated to just hosting Win32.

Does it support suid?

Um, I've had plenty of times I got a UAC (~windows sudo) prompt trying to access files I didn't have permissions to.

Doesn't WSL support Linux permissions via EAs? Couldn't they use the same model?

Not much honestly, NTFS is known for corrupting data over a long enough timeline (so it isn't suitable for server storage).

> NTFS is known for corrupting data over a long enough timeline

Any links?

Apparently it's called "bit rot": https://www.reddit.com/r/DataHoarder/comments/67h28o/bit_rot...

Bit rot has nothing to do with NTFS though? It's a general hardware degradation issue. (Where does that say it's NTFS-related?)

There are filesystems (eg. ZFS, BtrFS) which can detect and recover from hardware degradation. NTFS cannot, therefore the problem is NTFS-related, although not limited to NTFS.

If that's what is meant, it's sure as heck misleading to single out NTFS with "known for corrupting data" as if it's somehow unique in not having error correction compared to ext4 or whatever.

Error detection can be done in the layer underneath, including doing so within device mapper.

A filesystem not doing its own vs a filesystem "corrupting data" are very different concepts.

I have never heard that before.

Any business anywhere running Windows file servers is running NTFS for server storage.

Any business running a Hyper-V virtualisation, including large cluster, is running their virtual disks on NTFS. There's an irony here in Microsoft's next generation file system being "ReFS", with "resiliant" literally there in the name. And it has such a history of bugs and unimplemented features it's not safe for use outside of specific scenarios.

Edit: the link phendrenad2 posted has a comment about a "server destroying bug" in ReFS.

That thread [1] is 4 years old. Is there recent credible evidence of ReFS being problematic?

1. https://www.reddit.com/r/DataHoarder/comments/67h28o/bit_rot...

Microsoft's documentation still refers to recommending NTFS "for general use" and then list specific places it's designed for. Also listed features still not available often make it a show stopper:

https://docs.microsoft.com/en-us/windows-server/storage/refs...

Most Hyper-V admins will be familiar with this thread which is still completely accurate. This is relevant as it's "by design" and won't change or be fixed.

https://community.spiceworks.com/topic/2286588-please-do-not...

Here's a Veeam thread from 2019 talking about ongoing bugs: https://forums.veeam.com/veeam-backup-replication-f2/windows...

Being able to use long file names. The "256 bytes' ought to be enough" attitude hurts in this age of emojis.

I am running a dual-boot system with BTRFS on the linux side and WinBTRFS[1] to make the partitions readable from the Windows side. This works quite well (which means I never had any issue with it).

[1]: https://github.com/maharmstone/btrfs

Does this mean we can boot Windows and Linux from the same partition? Just point the EFI boot-loader at a different kernel image for each? Or does NTFS not support mixed NT-style and Unix-style file permissions on the same partition.

Or does the Linux kernel refuse to boot from NTFS, because Linux people hate anything with a Microsoft smell?

I think that, worst case, you could have a /boot partition on ext4, and then / could be NTFS. Although at that point maybe it's best to have just the /home running on NTFS.

I'm not too familiar with how EFI handles it, but I think that grub would need some NTFS support?

No need for hate. We just get grumpy when Microsoft enforces patents or calls Linux a virus, which they haven't done in a few years ;)

> I'm not too familiar with how EFI handles it, but I think that grub would need some NTFS support?

I use EFI, but I keep all of grub, the kernel, and the initramfs on the EFI partition. So grub wouldn't need to know anything at all about the partition for /, as it isn't loading anything from it.

I think you can do this today with some other filesystems. e.g. WinBtrfs is (rightly!) very proud of being able to boot a full Windows OS.

Oh wow, thanks for that, that's a cool project.

It's not really up to the kernel. The kernel and initial ramdisk are loaded into memory by the bootloader. Once you are in the ramdisk you can do pretty much whatever you want to mount the root partition; including using a user space file system. If you wanted to, you could skip switching to the "real" root filesystem, and run your system entirely out of the ramdisk.

With EFI, you can even skip having a seperate boot-loader entirely, and combine the kernel, ramdisk, and an EFI shim into a single EFI executable.

I guess that makes sense. If the kernel can mount an NTFS system without any external requirements now, then I guess it could set the fs root to that mounted volume.

Now all that's left is for Microsoft to add EXT support to NT.

There are third party tools to read from Ext partitions, but unfortunately, writing to the filesystem seems to be a footgun roulette situation.

Since they've went the way of having WSL2 physical mounts show up in \wsl$ this summer I don't think there is much push for native <pick your FS> support.

https://devblogs.microsoft.com/commandline/access-linux-file...

Would definitely be cool but Linux supporting NTFS only makes it even less necessary on top of that.

I hadn’t considered how this being in the upstream kernel would make it possible to run my WSL2 on NTFS natively. That’s pretty cool. I’m sure there’s still some stuff to figure out, but still pretty awesome.

Doesn't WSL2 already have the ability to read NTFS somehow? Isnt this just an optimization?

It’s a network mount. Hence it’s “don’t use it cause it’s shitty” reputation.

Well you could do that too but you'd lose permissions between the two (NTFS supports POSIX permissions but Linux doesn't and vice versa with Windows permissions).

The blog post is really about making any Linux mountable FS available to the Windows side of WSL2 without NT needing to support it. E.g. it doesn't matter if the drive is NTFS or Ext4 or BobYourUncleFS as long as the Linux kernel can mount the FS you can mount it in WSL2 and access it from Windows without NT needing to support the OS. It also already works in reverse, volumes mounted in Windows (including NTFS) can also be shared with WSL2.

It's really just their integrated-into-the-OS version of VM guest filesystem sharing.

What's the point, though? You still can't have two kernels accessing the same filesystem directly, so if you give direct access to Linux in WSL, Windows loses it.

You can only mount entire physical disks under WSL2 though, as opposed to single partitions, which makes it a showstopper depending on your layout. (Common single disk configurations like a laptop/ultrabook come to mind)

Looking forward to testing this out with games installed in my Windows partition. Hopefully won’t need to have a dedicated partition for Linux games anymore.

why couldn't you have used ntfs3g before then in read-only mode. or even the only ntfs in kernel code in read only mode.

Some games flat out won't work with read only and for the ones that do if they are still receiving regular updates it's going to be a PITA. Also a PITA when you want to add a new game.

Not to mention both ntfs-3g and even the previous kernel driver weren't exactly performance marvels.

I'd hazard to say that games that get updates wouldn't always work in a dual boot env either. i.e. if the update stores data in the registry / game depends on registry settings.

Is this as big of a deal as it seems?

Paragon's NTFS implementation is perceived as more mature than the alternatives. And since Windows + Linux on one system is quite common, this will make Linux more attractive for many Windows users.

Reply on news.ycombinator.com