Hacker News Reader

Comment by matja | Hacker News Reader

I understand the rationale of having a relatively simple filesystem specification so that it can be implemented under the constraints of embedded systems, but using a filesystem that has no journal or checksums on the cheapest and flakiest of storage devices (hi, bottom-of-the-barrel SD cards and USB memory sticks) seems a recipe for disaster in terms of data resiliency.

I use ZFS on SD cards and USB memory sticks when I am able to, and the number of checksum errors are ... not rare. Even for high-quality NVMe SSDs on (lower quality) USB-NVMe bridge controllers, corruption everywhere.

View on HN

The real issue that the article only hints at is Windows support. Windows only supports a small handful of filesystems (FAT*, ExFAT, NTFS, ReFS, ISO9660 & UDF - I think that's it!). So if you have to interoperate with Windows then you have to use one of those filesystems.

If you don't need to interoperate with Windows then you have many better choices.

> Windows only supports a small handful of filesystems ( [...] If you don't need to interoperate with Windows then you have many better choices

Are the filesystems choices mare varied if you need to operate with MacOS?

Whenever my dad brings me one of his FAT32 USB drives with "issues", I find a bunch of hidden dot files and folders that were not there before and realize it's been plugged in a Mac.

I really would like to know what MacOS does to USB flash drives.

The dotfiles are for metadata that the non-Mac filesystems can't store. Coloured labels, icon position, longer filenames, creator info etc.

Windows actually does almost the same thing. Thumbs.db etc. And it also creates shadow files for those with metadata when saving to FAT32. You just don't see them on windows as they are hidden, just like dotfiles are on macOS.

The last version of windows that made thumbs.db was XP.

And what metadata do you mean? Permissions and alternate streams get purged, don't they? What else is there? You have to go out of your way to get a desktop.ini and those are just as obtrusive on NTFS.

Long file names are stored in a weird way but that's very standardized and doesn't make extra files.

There's still the "System Volume Information" folder. Whatever that is. (I use Linux, but see it on USB drives that where plugged into Windows machines.)

On NTFS no, but Win 10 still makes shadow files on filesystems without metadata, like exFAT. Not sure about 11 actually as I don't use it much.

I forget the name but I believe it's the filename prepended or appended with an underscore.

PS: On Mac you can turn it off with a simple defaults write command. Of course it also means that metadata changes won't be persisted to such files. Not sure about Windows.

Are you sure? What metadata specifically? I've never seen anything like this from Windows.

And I think you misread my NTFS mention. That was only in the context of desktop.ini, and desktop.ini only happens if you go through a specific obscure menu.

But I can tell you that prepending filenames with ._ is a Mac thing.

hmm ok I was sure but you made me doubt myself now :) I'll give it a try tomorrow.

Windows 10 also creates a "Sytem Volume Information" on every USB drive.

> Windows then you have to use one of those filesystems.

I can appreciate that's true if the client systems are not under your control, but if you have influence over all systems that will touch the drives, I've had good experiences with OpenZFS which has drivers for all 3 major OSes: Linux, macOS, Windows

One just needs to be cautious not to use bleeding-edge features because the ZFS release cycles differ, but otherwise so far it has done fine for me

The system with Windows is more likely to be a company-owned computer, or a computer owned by some friend, acquaintance or colleague, on which you cannot install file system drivers, so exFAT is the best bet for compatibility.

If it is your own computer, then you either are a Windows user or you prefer other operating systems, in which case you would wipe Windows and install something else. In both cases you do not need to transfer data between operating systems, so you can use on external memories NTFS for Windows or XFS, ZFS etc. for Linux.

> If it is your own computer, then you either are a Windows user or you prefer other operating systems

That fits me, if it's inclusive or. Windows is a pretty OK desktop environment, and mostly gets the better tested mass market software. On the other hand, I prefer other operating systems for doing "real work". I don't usually need to transfer data via external storage though, but if I did, I might consider installing a filesystem driver/tool on my Windows machines to have a better time of it. OTOH, I might just use par2 or something on top of a readily available filesystem.

Actually FAT filesystems with two FAT entries can be implemented in a transaction safe way: 1) write allocations on first FAT, 2) write file data, 3) write the second FAT. This is the same as a journal that stores metadata only (what most other journaled filesystems do). This is probably what Microsoft calls "TexFAT", but doesn't describe in any way.

Good written data (not corrupted while being written) will last a long time, probably more than any magnetic media, so data resiliency is good.

> Good written data (not corrupted while being written) will last a long time, probably more than any magnetic media, so data resiliency is good

Are you meaning generally any kind of solid-state storage here?

I'm not sure about TLC and QLC, but SLC and MLC seems very good so far.

> will last a long time, probably more than any magnetic media, so data resiliency is good.

Flash storage will not last as long as magnetic media. You've got 50 years, tops, and even then it requires some luck to not have any bit rot.

> Flash storage will not last as long as magnetic media. You've got 50 years, tops, and even then it requires some luck to not have any bit rot.

Which magnetic media lasted 50 years? I don't belive HDDs from the '70s are readable today.

I tried to read a QIC-120 last year (2021). The tape was dated 1992 (I think), so it was almost 30 years old. The tape jammed inside the cartrige.

50 years, for flash ? without wear leveling ? Do you have some links ? Over 10 years for flash is difficult.

Doesn't most flash memory already have some builtin error detection and possibly correction logic already?

It is supposed to have, but apparently cheap flash storage does not have this logic, or it's not working. That's what GP is referring to.

Flash sticks are like $2 wholesale parts. The packaging probably costs more than the product in many cases.

Every expense has been spared - I wouldn’t trust these things at all unless you are using one of the more expensive SSD devices or something certified for Windows ToGo, etc.

All flash memory devices are wall-to-wall error correction. It's basically math on a stick. The raw media error rates are 1% to 10% which are crazy high. That flash works at all is a miracle of linear algebra.

You can also get ext4 (with journalling) rather small so it's usable on an MCU.

https://github.com/gkostka/lwext4

Reply on news.ycombinator.com