Comment by lsc | Hacker News Reader

Comment by lsc | Hacker News Reader

Parent Full thread

I'm looking at

http://www.supermicro.com/products/chassis/4U/847/SC847E26-R... (36 drives, and room for a motherboard)

or

http://www.supermicro.com/products/chassis/4U/847/SC847E16-R... (45 drives, das. I have had serious outages caused by DAS back in the SCSI-3 VHDCI days, so I'm likely to start with the 36 bay system.)

for my mass storage project. they are both well under $1500 with power supplies, backplanes and expanders taken care of.

For me, the big thing is that with my storage model, I'm going to be replacing disks as they fail and rebuilding the RAID, so having easily accessible and easily swapped disks is worth paying a premium. (I am planning to have some cross-chassis redundancy by using zfs snapshots, but I'd rather just keep the nodes going as is.)

Also, rack density? doesn't save you that much money. Most of what you are paying for in a data center is power. At the cheapest co-lo I'm in, here is the cost breakdown:

1 full cabinet (44u) with two twenty-amp 120v circuits: $875 1 full cabinet (44u) with one twenty-amp 120v circuit: $530

So, if I can double my density, I save $185 a month; and even at the disk density for my compute nodes (close to 100 disks in a rack) I get one disk failure maybe every two months per rack; so if I have to slide out the whole goddam computer, causing some chance of the power getting disconnected and downtime? yeah, with my model? it's probably worth paying the premium.

I'm just saying; if you are small enough that paying five grand for a backblaze pod.

Sure, at-scale, it's best to design your systems so that you can get zero downtime even when hardware fails. But, that's really difficult to do without introducing new failure modes; even amazon has trouble with it. My strategy is to accept that hardware failures mean a truck roll and downtime for customers on the hardware in question. As long as you don't have any one server go down more than once a year, (and a particular server failing once a year is pretty pathetic) and as long as you don't have a system where any one server brings down everyone, you are going to see pretty good reliability using this strategy.

This is a great thread here.

I think there are a lot of people, even institutions like where I work, that are looking hard at storage that isn't block based storage from some huge vendor or storage that ends up being $5000+/TB (looking at you Isilon). Ok, maybe it's just me all alone here at my work place. :)

Even running RAID6 or RAID10 over 3TB SAS drives on some largish enclosures (ie. 36,45,60 drives per 4U) will be very cost effective even just using md, lvm, and xfs.

Or even better, use ZFS if possible. Not a huge fan of Solaris, but there is OpenIndiana, or FreeBSD has ZFS.

And object storage such as openstack swift is really gaining momentum and will likely replace most storage systems for large amounts of data over the next couple/5 years. There are single orgs that have put out 5.5 petabytes of openstack swift storage! Right now!

Who would you buy through? Googling for the 45 drive version gives me slightly higher pricing.

Also- The big advantage of the BackBlaze version is that people have done it before, and it mapped out. With the SM case, there's less community.. But it does look like a good solution.

It looks like the drive bays are pre-wired up, so you wouldn't need to worry about that? What else would you need? Mboard/CPU/RAM, Raid cards, HDs and sleeves?

Like most supermicro chassis, it comes with the drive caddies, backplane, and power supplies. All you need is the motherboard, cpu, ram, and a SAS card, well, and the drives. It's even got an expander built into the backplane. If you want h/w raid, you need to bring that as well. (I plan on using zraid2, most of the raid cards that cost less per port than the drives are not better than software raid.)

I buy most of my supermicro stuff through kingstarusa.com - I know the site looks a little shady, and you have to email for quotes for almost everything, but they are good people. My office is actually above their warehouse; I'm unit C. Their price is usually a few dollars less than provantage, which is usually the next best retailer for supermicro chassis, and they don't do shady tax dodge bullshit, and I don't have to pay shipping. I could be misremembering on the exact price on the 45 bay.

Most of the 'mapping out' done with the backblaze version is already done and tested on the supermicro.

most of the raid cards that cost less per port than the drives are not better than software raid

Let's not get carried away; isn't a "gold standard" LSI controller only ~$1,000 ($27/drive)? But you certainly can buy a lot of Intel cores for that price.

yeah, I am exaggerating on the port cost some, but the big advantage of hardware raid over software isn't the hardware calculation of parity. A CPU can calculate parity so much faster than you can write to a drive that it doesn't matter at all that special hardware can do so even faster still.

The advantage of the hardware raid card is the battery backed cache. If it doesn't have a BBU and a fair amount of cache, as far as I am concerned, you might as well be using MD.

Hardware RAID cards have improved quite a lot recently; some of the stuff now has reasonably sized caches, so perhaps I should revisit my assumptions in this area. Of course, I'm planning on using ZFS on my storage servers, so even if hardware raid cards are now a reasonably good deal, they won't do me a whole lot of good.

As much as I like ZFS, I do feel a little misled by the rhetoric about replacing "expensive" BBUs with slog SSDs... that are actually much more expensive.

Until quite recently, I would not have understood what you meant. The cost per gigabyte for even really fast SSDs is lower than the cost per gigabyte of RAID cache ram, so I'd have said "what are you on about?"

But, I think I understand what you are on about now.

Most of us (well, speaking for myself, but I think this is true of most SysAdmins) have very strong experience telling us "more read cache is better" - I mean, more read cache, up until you can cache everything the server commonly reads, makes an absolutely huge difference in performance.

So we look for big caches.

The problem is that most of us don't have the same intuitive grasp of where the benefits stop coming when adding more space to the write cache, as most of us don't have a whole lot of experience with large write-cache systems (outside of netapp/emc type boxes, and I personally attribute their superior performance in part to their gigabytes of ram that can be safely used as write-back cache.)

So if write cache works the same way as read cache? yes I will pay the premium for the fastest 32GiB SSD I can find, if I can use it all as write cache.

The thing is, I'm told, that after a few gigabytes, the returns to adding more write cache fall off sharply; and if that's true, then yeah, you are right, 'cause you are wasting most of the SSD.

I mean, the real question here is "how much write-cache do I need before I stop seeing significant benefit to adding more write-cache?" and if that number is much above what you can get in a RAID card, then the zfs/ssd setup starts looking pretty good.

Backblaze designed their system to be redundant across multiple systems with the ability for one to go down seamlessly.

If losing one system from your network will cause unacceptable downtime then you shouldn't consider using Backblaze's design.

clearly. I'm just saying, it's much easier to design a system where hardware failures cause downtime, and where that downtime is made acceptable because hardware with redundant drives doesn't fail that often than it is to design a system where hardware failures don't cause downtime (and where a failed hard drive causes a full node failure)

Thus, to someone who isn't at scale, the supermicro systems are likely going to be cheaper, overall, than the backblaze pods. (It sounds like these people paid more per disk for the backblaze pods than I'm going to pay for the supermicro pods anyhow.)

I just built two of these 36 drive systems. $9k for 108TB raw. Feel free to drop me a line, and I will share any details.

sweet. what'd you use for your sas card?

I've also done three of these builds so far. Used the SC847A chassis with direct iPass cable access (i.e. no port multipliers), 4 drives per cable. Downside is you need 9x SFF-8087 connectors on controllers, and 9 iPass cables to somehow route. Don't get the SM iPass cables, TrendNet makes better ones. Upside is you have dedicated SAS2 bandwidth from the drive all the way though the controller and the PCIe bus. Likely overkill. Also a tip, you can mount 4x internal 2.5in or 2x 3.5in drives. SM has the part numbers for the brackets on the chassis' product page. Don't put anything you'd remotely want to hot swap in these brackets, they will be buried under the motherboard tray.

I've used 4x LSI Logic 9211-8i controllers plus the onboard of the SM X8DTH-6F. Both the onboard and the 9211-8i use the LSI 2008 chipset. I have Solaris and ZFS setting on top of these, so I don't have hardware raid.

This is actually very solid hardware so far. I had a PSU fail and thats it. Let me know if you have any questions.

Reply on news.ycombinator.com