Hacker News Reader

Comment by Animats | Hacker News Reader

"Causes non-serious data loss". What does that mean. Only affects cat videos?

I believe the Debian policy for reports says that for corruption you can recover from. Serious data loss would be losing your metadata and not being able to figure out where any of the files are, or corrupting all copies of the disk encryption key.

I think "corruption you can recover from" seems far too vague; you can recover from any corruption with backups.

The way I'd interpret it is that it's non-serious if fsck can repair it. Not sure if that's accurate though, just the way it sounds to me.

If you're trying to be padantic and missing forest for the trees... Sure.

Sure, and you can recover from any hardware failure with a separate computer. In fact, you can recover from death simply by cloning yourself. What's the issue?

The issue is that some people don't have the time, money, and/or willpower to invest in backups. Some people only have one volume, or one copy of each volume at least, and need to recover their data using only that one volume if something gets corrupted. That's what "corruption you can recover from" means. Using a backup is not recovering from corruption; it's replacing the corruption with the pre-corrupted data. Recovering from corruption involves actually transforming the corrupted data back into non-corrupted data.

I think you're agreeing with userbinator here.

Not really. They're arguing that backups count as recovering from corruption, where my argument is that backups don't count because the only thing you're doing is discarding the corruption, not recovering from it. You're basically throwing the corruption out and getting your data back from somewhere else.

So you recover your data, but not from the corruption. You recover it from a backup. Similar, but not the same thing.

In this case, I would say that backups are not relevant protection. In order to restore something from backup, you would first need to know that a restore from backup is needed, and secondly you need to know what the last-known good copy is.

This is a silent data corruption bug (data is written at the wrong offset, IIUC) with no known method to detect the error condition except to compare the resulting data with a known-good copy. The next backup run will happily accept the corrupted file contents into the backup vault, and unless someone happens to stumble upon the corrupted data, the last-known good copy will eventually be rotated out.

I think they meant non-catastrophic (no pun intended). From what I can tell, individual files can get corrupted when opened with O_DIRECT, but the filesystem structure will never fail.

From the link in https://news.ycombinator.com/item?id=38591444 :

> file position is not updated after direct IO write and thus [we] direct IO writes are ending in wrong locations effectively corrupting data.

This may be a major problem for hypervisors using ext4-backed disk images. It wouldn't surprise me if qemu opens image files with O_DIRECT by default, but it would surprise me if professional shops were using ext4 as the backing store rather than a provisioning abstraction layer like dm_thin. Still, anyone using kvm on top of ext4 must not be having a good day.

Reminds me of this exchange from Coupling (UK):

“How is it not serious if she died from it?”

“She made an unsuccessful recovery”

Funny, I also thought of Coupling, except I was thinking of

- how badly can you phrase yes?

[Cuts to other scene]

- No

Such a good show.

You lose your data but, like, it's just a prank man.

yeah, practical joke for using Linux...have a laugh and deal with it

Most data is recoverable albeit slow to do. It takes some very bad conditions or intentional actions to make data recovery impossible. Current high level standards for military/diplomatic data sanatisation are complete physical destruction of hard disks.

Even linux tools like shred have given up saying they can actually delete data from disks due to how SSD's work these days.

> Even linux tools like shred have given up saying they can actually delete data from disks due to how SSD's work these days.

Which emphasizes the importance of enabling full disk encryption immediately whenever you start using a new device--BitLocker if you're on Windows, FileVault on macOS, LUKS on Linux, etc. Trying to decrypt data is much harder than reconstructing deleted data on a stolen drive.

True, properly zeroing out the headers on an encrypted drive will make recovery impossible.

How to do that reliably is another question.

You cannot with normal tools as writing a 0 to the SSD does not guarantee it overwrites the 0 you want. At best it does, at worst it writes the 0 somewhere else and remaps the bit (or whatever its physical storage thing is).

This is especially a problem on macOS:

https://support.apple.com/guide/disk-utility/erase-and-refor...

> Note: With a solid-state drive (SSD), secure erase options are not available in Disk Utility. For more security, consider turning on FileVault encryption when you start using your SSD drive.

So if you set up a Mac without FileVault you can never erase everything.

At least with my Lenovo I can do the secure erase.

SSDs are not magic. For shredding to be impossible the drives would have to have hidden capacity (and a lot of it, more than 1%). People often say, write levelling makes overwriting useless, but what if you overwrite the entire disk? How would you recover after something like this?

Of course providing something like 10% of hidden extra capacity would extend the life of the drive significantly, but are manufacturers really doing that and are not mentioning it in their marketing materials? I never heard that they do that.

Sure, but very few business tools are doing that, they are playing the old Gutmann game which he himself has walked away from as not reliable anymore. Filling up the drive is certainly getting you somewhere compared to some of the snake-oil out there. Drives are only getting bigger though and rarely filled up, done some of this stuff in my time and if cost is no barrier then you'd be surprised how much can be reclaimed. Degauss and destroy is the only real method we have now and that's probably not changing in future.

>but are manufacturers really doing that and are not mentioning it in their marketing materials

SSD manufacturers have been caught out repeatedly with their in-built deletion API claims. Recovery of a significant % of files is possible nearly always without tampering having occurred.

> Of course providing something like 10% of hidden extra capacity would extend the life of the drive significantly, but are manufacturers really doing that and are not mentioning it in their marketing materials? I never heard that they do that.

Every SSD has considerably more physical blocks than reported blocks. They have to, SSD bad blocks are common and number of writes is quite limited compared to hard disks.

https://download.semiconductor.samsung.com/resources/others/...

> I never heard that they do that.

Not sure how you missed it, as it's not a new thing. Straight from the horses mouth, if that helps: :)

https://www.kingston.com/en/blog/pc-performance/overprovisio...

Note that it's not just a Kingston thing, it's common across SSD manufacturers. Or at least with the major ones.

SSDs don't cycle blocks based on how much you've written. You can write over the entire reported capacity of a drive 100 times and there's still the chance, however small, that your encryption key and some encrypted data are still sitting around on some chip. You just can't guarantee that it's gone. For all you know, the controller has marked them as bad and isn't going to reuse them, even though there's still recoverable data on them. Since there's no way to reach them through the drive interface, there's also no way to erase them. But if someone opened up the drive and pulled the chips, there they would be.

> what if you overwrite the entire disk

People who want secure deletion generally want their storage device to remain functional afterwards. Same reason they don't simply destroy the disk to get rid of its contents: those things are expensive. Overwriting the entire SSD every time you want a file gone just isn't a solution.

I don't agree - modern NVME drives have secure erasure mechanisms. All data is by default encrypted on-the-fly within the flash memory, and when requesting a secure erase, the controller throws away the key and generates a new one.

https://man.archlinux.org/man/nvme-format.1

In "reduce, reuse, recycle", this is the #2 R

Please don't trash physically good drives with a hammer. It's not good for the environment (or the drives!) when you have such a simple technology at your disposal!

What percentage of people on Earth use this in your expert opinion?

Also have you done any work in digital forensics?

Eagerly awaiting your response.

> I don't agree - modern NVME drives have secure erasure mechanisms.

Assuming the firmware doesn't lie about it does:

* https://www.zdnet.com/article/flaws-in-self-encrypting-ssds-...

So you’re a dog person?

I thought we all were dog people. Woof.

Some of us are different kinds of animals.

Some of us are just dogs.

On the Internet, nobody need ever know.

P.S. Squirrel!

Reply on news.ycombinator.com