Hacker News Reader

Comment by hyperman1 | Hacker News Reader

One reason: If you send a flush command to a consumer grade hd, it returns before the flush is done. This makes it look nicer on older HD benchmarking software. Enterprise hds return when the flush is complete, so databases acknowledge a commit only when it was actually succesfull.

View on HN

I think your wires may be a little crossed. This is unlikely to have ever been true for magnetic drives, but there is a quite significant grain of truth (no pun intended) to it for SSD.

Consumer SSDs may indeed return an early indication that a flush is complete, but whether they are lying depends heavily on your notion of persistence:

- the SSD may return done once the data is buffered in RAM

- or once it has reached a temporary log

- or once it has reached a 'permanent' location on flash

- or once it has recorded the mapping of data<->location to some separate directory

- or once it has updated some separate root pointer pointing to the latest version of the log/mapping structures in some separate special storage

I don't believe any SSD waits until the last step above before returning done. Separately, the SSD:

- may have no electrical power outage protection whatsoever

- may have supercapacitors fitted

- the supercap may guarantee full durability of every acknowledged flush

- or only guarantee atomicity up to some prior acknowledged flush

- or only guarantee that the drive has /some/ data on it in /some/ state on the next reboot

SSDs are insanely complex

It is interesting that to remain competitive in benchmarks, SSD vendors have to sell you an incredibly complicated software system alongside their flash chips soldered to a circuit board. They can't fix your filesystem, because you won't switch (and why pay them when your OS comes with one for free). They can't fix your database, because you won't switch (you already pay Amazon a million dollars a month for it). So they have to implement all the complicated parts of a database or filesystem, then try to understand the stream of events that real databases and real filesystems send it, and then restructure them to win the benchmark. All of this with no source code and probably no real test suite, and it all ends up being slower than a database that could just write to the flash chips directly.

I am also "excited" when I think about how the SSD shards blocks across flash chips and adds erasure codes, then people put the SSDs in RAID to shard blocks across SSDs and add erasure codes, then people put a filesystem on top of that that shards blocks across arrays and adds erasure codes, then put an application on top of that that writes to 3 replicas and writes erasure codes to 2 more. Now your "hello world" text file uses 8 gigabytes of storage, but at least you only lose it if there's one bug in the SSD controller, your RAID software, your filesystem, or your application. Or a tornado hits the SSD. Oh well.

HN often contains articles like "abstraction is bad", but even their contrived examples aren't this scary. I'm still pleasantly surprised when I log into my computer in the morning and still have a .bashrc.

I'd been hearing rumors (possibly spread by the mfgs to scare enterprises away from cheap IDE drives) about it during the late 90's and early 2000's.

Then bfitz proved in in 2005, and it turns out SCSI drives sometimes do it too: https://brad.livejournal.com/2116715.html

Oh no, this one came from I think the win98 era. I read about it in some (paper!) computer magazines, I had no or barely internet at the time. SDDs had not yet been invented.

Once one vendor decided to screw us, the others had no choice. It was trivial for benchmarks to work around it, too: Just send more data so the cache becomes irrelevant. The whole episode took a few months at most.

I dont remember if they flat-out ignored the flush, or started a flush but didn't wait for completion.

How can I find out which drives exactly have synchronous flush and which don't? Is it a widely publicized fact?

> If you send a flush command to a consumer grade hd, it returns before the flush is done.

So essentially its a no-op?

Just speculating but perhaps the consumer ones return after the cache is populated and not after the data is durably on disk? That wouldn’t really be a no-op but would be much quicker for small writes at the expense of correctness in a power outage (although in the case of a DB the WAL entries are typically atomic so it would just be a rollback anyway).

Think of it like mongodb thinks of data: it's eventually consistent, maybe.

source? this sounds like an urban myth. there's no way that every single consumer grade hard drive "returns before the flush is done", given that: 1. consumer grade is not well-defined, 2. there are many different "flush commands", and it does not depend on whether a drive is "consumer grade", and 3. if this were the case, then filesystem and database corruption would be rampant, which it is eminently not.

it would be impossible to write a remotely reliable high-performance database or filesystem if flushing were useless. professionals doing formal analyses like https://www.sqlite.org/psow.html would be a complete waste of time, because everyone would be just praying anyways.

>> consumer grade is not well-defined WD has laid out a reasonable standard. Blue/Black/Green are consumer grades. Red is for NAS and Gold is for enterprise.

https://brad.livejournal.com/2116715.html

some enterprise SCSI drives too. This was all back in the late 90's/early 2000's, no idea if it still happens with SATA or SAS drives...

sure, and "consumer" filesystems/databases were also pretty bad. ext3 was only released in 2001, and Windows supported FAT system disk since at least Vista. the "standard" OSS database was MySQL until at least 2010, if not later. I argue 2020 is different though. despite the onslaught of mongodb 'engineers', data reliability is far better than it was in 2000.

Reply on news.ycombinator.com