Hacker News Reader

Comment by infogulch | Hacker News Reader

I agree, there is only one reasonable method to consider a drive "wiped": encrypt all data written to it with a key; wipe == destroy the key. Drives are basically little computers now, they have full-featured CPUs, OSes, memory, write caches and persistent buffers, they compress data before writing it, lie to you about where the data is (ok fine, it's a legitimate abstraction), and who knows what else. My point is that drives have become way too complex for wiping by "overwriting" "every" "block" to ever be an option again (and yes all three of those words deserve scare-quotes). The answer to this problem has become: always encrypt, lose the key to wipe. Period.

View on HN

From your comment I'm not sure if you're aware but there are two established standards that do just that:

ATA Secure Erase is an optional security feature implemented by many flash-based disk devices specifically because write degradation and overprovisioning preclude them from being wiped via overwrite as you would magnetic media.

More SSD disk controllers than you may realize have an AES encryption engine and persistent storage for the key. The controller is encrypting/decrypting on the fly transparent to the user/system. A secure erase simply rolls that key (and flags all the cells as empty).

As noted in the sibling comment, there must be trust in the device itself that the old key is unrecoverable (or the data recoverable if the prior key had been compromised).

hdparm on any linux liveusb (e.g. Clonezilla) and an eSATA dock and you can nuke a few disks a minute with two commands:

  hdparm --user-master u --security-set-pass p /dev/sdX
  hdparm --user-master u --security-erase-enhanced p /dev/sdX

hdparm has other arguments that can tell you if this is supported and how long the drive says it'll take to do it.

The other standard is TCG's OPAL. An AES media encryption key (MEK) is generated to encrypt/decrypt all data on the fly. The MEK is encrypted with a user-defined key encrypting key (KEK) and that encrypted MEK is stored on the drive. At boot the user (or some external process) needs to supply the drive with the KEK to decrypt the MEK and begin transparently decrypting/encrypting data on the fly. Splitting the disk into multiple "locking ranges" with independent MEKs and KEKs is also described in the spec enabling secure wipe of just a portion of the disk. Lose the KEK or destroy the MEK and the data is unrecoverable.

I'd say this option still requires trust in the disk itself but since TCG dictates not just the interface but also the implementation and has a corresponding certification process (as well as reliance on an off-device key), I reckon TCG OPAL devices should be more trusted than those that simply claim to implement the ATA Secure Erase command, but I haven't had an opportunity to test any.

Cool, thanks for sharing.

It's interesting that even the drives themselves use this strategy to "erase" data; it doesn't actually overwrite the cells, it just always encrypts all data that goes to cells and rotates the key when it's asked to erase. This strategy is clearly superior, it doesn't cost any wear on the cells to perform, and it's instant.

The multiple locking ranges is pretty neat, I'd never heard of that.

How can one trust it though? Presumably the key has to make its way into the drive to actually decrypt the data. How can one be sure the key was actually deleted and not archived somewhere in the controller’s storage?

Ideally, data is encrypted at the CPU (or coprocessor or TPU chip) and sent to the drive encrypted. The drive never gets the key, its only job is to store encrypted blocks correctly. In the TPU case, the TPU chip controls decryption and the key never leaves the specialized, tamper-resistant chip; wiping means sending a command to the TPU to rotate keys, thus instantly destroying any access to the data.

A coworker once insisted on fondling discarded drives with extremely strong magnets, then taking a power sander to the magnetized part. I wasn't sure if this was overkill or not. Is there any wisdom in this?

Sounds like someone needing a reason to use a power sander

Yup exactly why I needed to buy my Milwaukee right angle die grinder… ask my wife ;)

Well if they wanted to be 100% certain nothing could be recovered, then that is a very good way to make sure.

When I worked for the UC Observatories we physically destroyed most retired drives, just in case of sensitive information. Some personal machines' drives were excepted but there was a good deal of hammering.

what if you just fill the whole disk with random bytes? Wouldn't that take care of it ?

There could be bits of plain text data left over in over-provisioned storage set up in case a block goes bad (e.g. a flash cell that can no longer erase).

Would that extra cell be used unless there is a need for it? If there is a need for it, then I assume it will be a need for it when you fill the drive too.

But yeah, it depends on the manufacturer of the drive, so you don't want to take the chance, Full encryption is unbeatable.

Flash devices maximize the longevity of the cells by wear leveling. The on-board controller maintains a map between the physical cell layout and the logical layout that's changing with every single write operation.

Extra cells are used both reactively and proactively.

Reactively: Error when erasing or write/verifying a cell? Copy contents to spare cell, flag old cell as bad, update the map, maybe complain if running low on spare cells but otherwise call it a day.

Proactively: Cells are only rated for so many erase operations which directly informs the longevity (total bytes written) of a drive. MLC cells can be written to 3,000 times and SLC 20,000 but the latter is much more expensive. For use cases where greater longevity is required (without spending SLC money), why not put a disk together that has twice as many cells as it needs and wear level across all of them? If you `dd if=/dev/zero of=/dev/sdX bs=64k status=progress` your 32GB drive and it'll complete after 32GB but you have no way of knowing how many more cells that drive has that you didn't touch.

(Also as one of the parents pointed out, disk controllers are getting smart. You don't know if it was transparently compressing data too causing you to touch even fewer cells. ATA Secure Erase is the way to go.)

You don't have access to "the whole disk". What the drive presents to the OS as an array of blocks is just an abstraction now. There are at least 3 commonly known ways that "filling" the disk with random bytes could fail to destroy the original data:

* If any part of a block is found to be corrupted for any reason, the block is transparently set aside and never used again, but its contents may still be present to read for someone that is willing to bypass the drive's firmware.

* To prevent uneven wear, ssd capacity is over provisioned which means it has more cells than what it tells the OS. In general, when you "overwrite" a block on your drive, that data likely does not actually overwrite the cell where the data used to be. The drive picks some other unused cell, writes the data there, and does some bookkeeping so it knows that this new cell is what the OS means when it asks for that block again. When the OS "fills the disk" it may not end up overwriting some of those cells due to the firmware's arbitrary wear leveling algorithm.

* Many drives have a local write buffer that uses different, faster persistent storage to temporarily save writes to the drive as it takes its time to write out cells. This is to prevent data loss in the case of a power outage. In case the drive hadn't fully written all the blocks to the main storage when it boots back up again, it finds the data in the write buffer and finishes the job. The write buffer has both of the previous issues except its even more arbitrary and depends on load profile which your random filling routine is not guaranteed to hit.

There is no such thing as securely overwriting blocks anymore, because "blocks" are a fiction presented to your OS by the drive's firmware. There is only one way to be sure the data on the drive is filled with random data: make it random to begin with, and ensure that it stays random by destroying the decryption key.

>* If any part of a block is found to be corrupted for any reason, the block is transparently set aside and never used again, but its contents may still be present to read for someone that is willing to bypass the drive's firmware.

How useful is one block of recovered data?

Blocks can be up to 4mb. You could store all your passwords, private keys, contact/identity info, etc

Really now. Last time I cared to look at any of this, it was all 4K blocks.

You are confusing physical blocks (i.e. flash memory) with logical blocks (i.e. filesystems).

Edit: HDDs typically have 4K blocks though ("advanced-format drives"), but we're not talking about HDDs.

Yeah, I honestly have not cared about how hard drives work in about 10 years since I built my last array. I have better things to do with my life than worry about this stuff. I just want to connect a drive, reformat/partition if necessary, and then go back to work. A single SSD now read/writes faster than a 16 drive array (got close in RAID0, but that's just dumb). Now, if SSDs could just get a decent volume size.

They've got what I consider decent size, it's just that the price can be a bit steep compared to an HDD, at least at quantities of 1. $700 will get you an 8TB SSD.

8TB at blazing speeds to boot in a single device. To get decent speeds from HDD would require at least 8 drives striped together. So not only do you need 8+ HDD drives, you also need the enclosure. That would easily double the $700 for the single SSD. Luckily, with USBC/TB4, these enclosures come with the controllers built-in instead of requiring PCIe board

Reply on news.ycombinator.com