Hacker News Reader

Comment by NovaS1X | Hacker News Reader

Impressive. First question that pops into mind is how long the rebuild time would take if one of these failed in a RAID. I can imagine it'll take a while.

View on HN

It takes 8 hours 20 minutes to write 15TB over 500MBps. Why would you want to stick one of these in a RAID though?

..it would be more like Redundant Array of Prohibitively Expensive Drives.

(I'll show myself out)

article says 1200MBps.. so more like 3.5 hours.

I wonder how that 1200MBps will hold up with the thermal throttling with their other V-NAND products (SM951/950 Pro), especially when doing a raid rebuild?

I happen to have a 950 Pro, and normal usage keeps the temperature at reasonable levels (though I stuck heatsinks on mine), but this is much more dense (512 vs 2 on the 950 Pro).

FusionIO drives (PCIe attached flash storage) have serious heat sinks. The heat sinks are not for show.

I get about 2GByte/second sustained. It's fun :)

i remember the times when RAM of this speed was something to brag about... guess i'm getting old.

RAPD is a catchy acronym.

You mean RAPED?

If you use a fully distributed storage system, like Ceph (or 3Par?), then rebuild times during a drive failure would be significantly lower. Also, any self-respecting storage system (RAID or not) will lower rebuild priority over normal IO.

To give a specific example, let's say you have a 10-node Ceph cluster with 10 of these SSDs in each node filled up to 75% capacity. During a rebuild you throttle down to 50 MBps per SSD. After a single drive failure, the remaining 99 SSDs will work together to redistribute the data in about 40 minutes:

  $ python3 -c "print(round((15.36 * 1024 * 1024 *.75) / (50.0 * 99) / 60.0, 2))"

40.67

The 50 MBps per drive may seem like a low number, but that actually means each node has to move data at roughly 5 Gbps over the cluster network.

* Edited to fix the python line where the multiplication characters converted to italics formatting.

Isn't 50MBps closer to 0.5Gbps than 5Gbps? Where does the other factor of ten come from?

> 15.36TB > sequential read and write speeds of up to 1200MBps

~4 hours to read/write a whole drive ain't too bad. Of course there might be other bottlenecks in your raid, or you only reserve so much speed for rebuild, but the drives size is not the problem.

I'm kind of shocked that this drive is so slow. That's an impressive benchmark compared to your run of the mill consumer drives, but the new PCI-e drives Apple has been shipping are already beyond 2000Mbps.

http://blog.macsales.com/30725-owc-tests-speed-of-ssd-in-201...

I would've expected Samsung to be able to best that. They make the chips, after all.

I thought RAID and SSD were a terrible combination. As if both drives do exactly the same writes at the same time, chances are they will fail at the same time. Plus most RAID controllers don't support TRIM.

Has the thinking changed?

"I thought RAID and SSD were a terrible combination. As if both drives do exactly the same writes at the same time, chances are they will fail at the same time."

This is a very good thought and you should be thinking it.

However, it is really only applicable to raid mirrors. Raid stripes will not have identical wear lifespans.

All of our mirrored boot devices are either:

- current intel enterprise SSD paired with the previous generation intel enterprise SSD

or:

- current intel enterprise SSD paired with current samsung enterprise SSD

That way if there is a usage related failure (or some weird firmware bug triggered by use pattern) they don't seize up identically.

> I thought RAID and SSD were a terrible combination

Well there are different RAID settings. My product is an indexing engine (indexes Git repositories) and I've personally found RAID 0 reduces indexing time by about 30% compared to a single drive. I have a machine that has 4 SSDs running RAID 0 and I've found the performance gain after 3 SSDs is negligible.

It seems like 3 SSDs running RAID 0 is the best combination given my very limited sampling size.

An SSD is implemented a little like a RAID array internally. Higher capacity SSDs have more flash chips and tend to be faster. So one would expect that 2 x 128GB drives in RAID 0 to perform similarly to a single 256GB drive.

Wouldn't being able to offload to multiple SSD controllers potentially help too if that's the bottleneck?

It would - you'd have the aggregate bandwidth of the multiple controllers to use. Similarly, RAID-1 is better for random reads than RAID-0.

> As if both drives do exactly the same writes at the same time, chances are they will fail at the same time.

You mean due to exhausting their endurance? This is something you monitor, you should have plenty of time to replace the drives before it becomes a concern.

For other failures, how's it going to be any different from normal HD's? There's always the risk that having the same models/batches in the same conditions might lead to a cluster of failures, but RAID's still likely to save you from plenty of other failure modes.

For instance, the recent Google study[1] suggests that uncorrectable errors with SSDs are actually more common than HD's: "More than 20% of flash drives develop uncorrectable errors in a four year period... significantly higher rates .. than hard disk drives".

I've certainly seen enough IO errors from SSDs to know I want to defend against them. Not to mention the silent data corruption I've seen (and been protected from, yay ZFS).

1: http://0b4af6cdc2f0c5998459-c0245c5c937c5dedcca3f1764ecc9b2f...

"You mean due to exhausting their endurance? This is something you monitor, you should have plenty of time to replace the drives before it becomes a concern. For other failures, how's it going to be any different from normal HD's? There's always the risk that having the same models/batches in the same conditions might lead to a cluster of failures, but RAID's still likely to save you from plenty of other failure modes."

He's talking about something else. Physical drives fail due to physical reasons, but SSDs fail for logical reasons.

A weird firmware bug or an unexpected usage pattern or wear pattern (not just burning the entire things lifetime up) can cause the SSD to die.

If you have a mirror, and you have two identical SSDs, and you produce such a condition ... then no more mirror. They both die identically.

See my response to the parent about how you can easily address this by mixing SSDs in mirrors.

> Physical drives fail due to physical reasons, but SSDs fail for logical reasons.

HDD's also have a long and distinguished history of nasty firmware bugs, and this is only going to get worse as things like drive-managed SMR and hybrid flash caches get more common and their internal complexity ramps up.

Both also fail due to electrical reasons. Chips fail due to manufacturing defects, solder degrades, capacitors dry out, etc.

And I'll reiterate the uncorrected bit error rate. Transient errors are more common with SSDs, and are unlikely to happen identically with multiple units.

> If you have a mirror, and you have two identical SSDs, and you produce such a condition ... then no more mirror. They both die identically

I have a pair of mirrored SanDisk Extreme Pro's in my main server. Both suffered from a firmware bug that caused data corruption. ZFS was able to repair all damage, because they didn't fail identically.

Also thanks to the mirror I was able to upgrade the firmware without taking the server down.

Mixing different SSDs might be a good idea, but you can make much the same arguments for doing the same with HDD's, and like with HDD's it's still better than nothing to have redundancy with "identical" drives.

I am actually referring to endurance. Irrespective of the RAID mode, I would expect each drive to be written exactly the same amount of data at the same time. If all models are identical and have been purchased at the same time, one would expect that the same firmware will allocate the same writes to the same cells (in RAID 1 it will be the same data, in RAID 5 the data will be slightly different, but the size will be the same). So in theory the wear should be identical. Obviously this only applies to a RAID mode where you have parity.

But I read that like 3-4y ago. We should have more experience with SSDs in production now so I was wondering if that thinking still applied.

> If all models are identical and have been purchased at the same time, one would expect that the same firmware will allocate the same writes to the same cells

Not really, you'd expect them to diverge over time as the cells aren't going to be all 100% identical - they'll have different failures, different error rates, and different read patterns will result in different read-disturb errors, all of which will effect even completely deterministic block allocation.

Regardless, as I said, you monitor endurance. Drives are unlikely to just silently wear out out of the blue unless you've been completely ignoring their SMART readings, in which case, more fool you.

Yeah, the statistical chance of the same thing happening on both drives is the same, but the chance of it happening at the same time is pretty low.

That sounds perfectly reasonable, and 20 years ago I'd have said the same thing. It's even fairly likely that I did say it.

You know where this is going, right?

I was a SysAdmin at the time and one of the things I was responsible for was Oracle Financials. It was at the core of everything that mattered at that company -- and I was absolutely convinced that short of a water balloon fight in the machine room, my raid configuration was 100% reliable.

The first disk died late on Friday. I had two on-hand so I casually replaced it -- and before I got back to my desk another had died.

I placed an order for more drives and went back to the machine room and replaced it.

By Monday morning, things were starting to get serious -- I had to drop back to raid 5 because a few more had died over the weekend and my replacements wouldn't arrive till Tuesday.

You see, all the drives in all our raid enclosures had come from the same batch -- and they all -- every single one of them -- died within 90 days of the first one.

The chances may have been low, but reality has sharp teeth and loves the taste of overconfident sysadmin ass.

> The first disk died late on Friday. I had two on-hand so I casually replaced it -- and before I got back to my desk another had died.

That happens very very very often. It has nothing to do with a bad batch.

The second disk actually failed a while ago, but no one noticed because no one read from that part of it.

When you did the rebuild you read from the failed area and woke up the failure.

When you setup raid you MUST read the entire raid at least monthly, so that any errors are detected early! This is absolutely critical. Without that you have not installed the raid correctly. mdadm on debian does that by default. Linux has a builtin way to do that, but you must have a tool that will alert you on failure, or it's worthless.

You should also run a week full disk read of each hard disk in the array using its onboard long self test feature. You can use smartd to schedule that automatically. And more importantly: Notify you on failure.

Not using both tools is not setting up the raid correctly.

I think you have different definitions of failure. The previous person seems to claim it meant the disk was dead (i.e. no read works) while you seem to claim that it means an error caught by low level formatting.

Those scrubs do nothing to catch errors that the drives do not report such as misdirected writes. Consequently, there is no correct way to setup RAID in a way that makes dats safe. A check summing filesystem such as ZFS would handle this without a problem though.

> The previous person seems to claim it meant the disk was dead (i.e. no read works)

He made no such claim.

> while you seem to claim that it means an error caught by low level formatting.

A: There is no such thing as low level formatting in a modern drive, and B: No I don't. I said he should do a full disk read. Not format.

The SMART built in self-test does a full read of the drive, not write.

> Those scrubs do nothing to catch errors that the drives do not report such as misdirected writes.

That's only true with RAID 5. Ever other RAID level can compare disks and check that the data matches exactly. The Linux md software RAID does that automatically if you ask it to check things, and then it will report how many mismatches it found.

If you look he wrote: "I had to drop back to raid 5". He had a better level of RAID before, with multiple disk redundancy, that allows the RAID to check for mismatches and even correct them.

But because he never scheduled full disk reads the RAID never detected that many of the drives had problems.

> Consequently, there is no correct way to setup RAID in a way that makes dats safe.

That is not correct. The only advice I would give is avoid RAID 5. The other levels let you check for correctness.

> A check summing filesystem such as ZFS would handle this without a problem though.

Only if A: you actually run disk checks, B: and only if ZFS handles the RAID!!! ZFS on top of RAID will NOT detect such errors 50% of the time (randomly depending on which disk is read from).

Low level formatting exists on modern disks. You just are not able to reformat it. There is a diagram showing it here:

http://www.anandtech.com/show/2888

Doing a full read causes every sector's ECC in the low level formatting to be checked. If something is wrong, you get a read error that can be corrected by RAID, ZFS or whatever else you are running on top of it provides redundancy. Without the ECC, the self test mechanism would be pointless as it would have no way to tell if the magnetic signals being interpreted are right or wrong.

As for other RAID levels catching things. With RAID 1 and only two mirrors, there is no way to tell which is right either. The same goes for RAID 10 with two mirrors and RAID 0+1 with two mirrors. You might be able to tell with RAID 6, but such things are assumed by users rather than guaranteed. RAID was designed around the idea that uncorrectable bit errors and drive failures are the only failure states. It is incapable of handling silent corruption in general and in the few cases where it might be able to handle it, whether it does is implementation dependent. RAID 6 also degrades to RAID 5 when a disk fails and there is no way for a patrol scrub to catch a problem that occurs after it and before the next patrol scrub. RAID will happily return incorrect data, especially since only 1 mirror member is read at a given time (for performance) and only the data blocks in RAID 5/6 are read (again for performance) unless there is a disk failure.

There is no reason to use RAID with ZFS. However, ZFS will always detect silent corruption in what it reads even if it is on top of RAID. It just is not guarenteed to be able to correct it. Maybe you got the idea of "ZFS on top of RAID will NOT detect such errors 50% of the time" from thinking of a two-disk mirror. If you are using ZFS on RAID instead of letting ZFS have the disks and that happens, you really only have yourself to blame.

Is there a well known tool to check for this?

If you are using software raid on linux then:

1. install mdadm and configure it to run /usr/share/mdadm/checkarray every month. (The default on debian.)

2. have it run as a daemon constantly monitoring for problems. (Also the default on debian.)

3. test that it actually works by setting one of your raid devices faulty and making sure you get an instant email. A tool that detects a problem and can't tell you is quite useless.

4. install smartmontools and configure /etc/smartd.conf to run nightly short self tests, and weekly long self tests. Something like: /dev/sda -a -o on -S on -m email@domain.com -s (S/../.././02|L/../../6/03)

5. do a test of smartmontools by adding -M test to the line above to make sure it is able to contact you

This way you will find out about problems with the disk before they grow large.

There are other settings for smartmontools to monitor all the SMART attributes and you can tell it when to contact you.

6. Extra credit: Install munin or other system graphing tools and graph all the SMART attributes. Check it quarterly and look for anomalies. Everything should be flat except for Power_On_Hours.

For btrfs, you'll want to schedule a scrub on a regular basis. This will also detect read errors and try to fix them.

If you use 3ware/LSI/Avago (now all Broadcom) you install the Megaraid tools and it has a scheduler that does a patrol read whenever you schedule it to. ReadyNAS devices have a similar setting that will check the disks.

I wonder, for such mission critical data a good strategy would be to start rotating out old drives at set periods. Maybe replace one drive of your RAID5 every 6 months, not matter what the health. Once you've replaced all the drives, they will all be staggered in age by 6 months. Hopefully then the chances of multiple failures is greatly reduced.

Replacing a drive in a RAID5 would mean voluntarily placing yourself in a high risk position every 6 months. If you're going to do that, you better make sure you use a raid level that gives you at least two drive redundancy.

In the GGP post he mentions being "down to just RAID5", so presumably he was at RAID6.

(Bookmarking this entire thread for much RAID wisdom.)

That would make your failure rate higher (from increased infant mortaility) and your maintenance more expensive (more spares, more transaction costs, more work, replacing perfectly fine working parts too soon).

Drives have on-condition monitoring via SMART so you can predict age related failure.

Scheduled replacement is almost the worst maintenance policy.

For mission critical data, you should have backups and use at least triple mirrors or raidz2. Quadruple mirrors or raidz3 are even better for the extra paranoid.

That might have been the case with HDD, and the rationale of using RAID. But I thought that with SSD, if you have drives in RAID 1 or RAID 5, each drive makes the same amount of writes at the same time, and if they are the same model / age, the firmware will allocate the writes to the same cell. And you would end up with exactly the same wear on all cells. And therefore all drives failing simultaneously.

Software RAID or a host-based controller is the way to go with SSDs. Otherwise the raid controller becomes your bottleneck as whatever onboard ASIC or RAID-on-a-chip hits its throughput limit (and potentially overheats and dies if it's some crappy product designed for HDDs).

TRIM is well supported by modern host-based controllers designed for SSDs.

I just (like, about the time you were writing your comment) replaced a pair of spinning disks with a pair of 120 GB SSDs in one of my machines here at home. It's not technically RAID but a ZFS mirror instead. The operating system and related files will live on it while my data still lives on some spinning disks (simply because of the amount of data).

I've got pretty much the same setup (albeit with several more spinning disks) in a bunch of servers and have yet to have any problems (* crosses fingers *).

A Raid 5 could work, since you are not writing the same data. You would need a hardware coprocessor to handle generating the checksums at 4GB/s though.

>You would need a hardware coprocessor to handle generating the checksums at 4GB/s though.

A first gen i5 2.4GHz can generate XOR for RAID5 at 4GB/s. Newer generations at faster core speeds should be able to beat that significantly.

...

I went and tested this on a Intel(R) Xeon(R) CPU E3-1220 v3 @ 3.10GHz

[1053854.445872] avx : 17860.000 MB/sec

[1053854.475878] raid6: sse2x1 gen() 6554 MB/s

[1053854.492857] raid6: sse2x2 gen() 9070 MB/s

[1053854.509853] raid6: sse2x4 gen() 11273 MB/s

[1053854.526848] raid6: avx2x1 gen() 21554 MB/s

[1053854.543844] raid6: avx2x2 gen() 25152 MB/s

[1053854.560838] raid6: avx2x4 gen() 29167 MB/s

[1053854.560839] raid6: using algorithm avx2x4 gen() (29167 MB/s)

[1053854.560840] raid6: using avx2x2 recovery algorithm

RAID 5 is terrible with 4K random IO. I know guys who measured >1 million IOPS with MD RAID 0 and then switched to MD RAID 5 only to have performance drop from read-modify-write overhead. They could not believe that RAID 5 was the cause.

They'll have the same lifespan, but they won't fail at exactly the same time. Also, a lot of RAID now is done in software, and most software raid types support TRIM.

Reply on news.ycombinator.com