135TB for $7,384 - Backblaze Pod 2.0
zdnet.com
zdnet.com
They got rid of the PCI bottleneck by switching to PCIe, a bottleneck which surprised me when they designed version 1.0 of their pod. They could have gone PCIe at the time, I maintain http://blog.zorinaq.com/?e=10 and they were SATA controllers at the time that met their technical requirements (nr. of ports, Linux support, etc).
"In the first generation storage pod, we ran out of the faster PCIe slots and had to use one slower PCI slot, creating a bottleneck."
Other than this, the other parts are pretty standard and are easy to find with some simple searching.
Feels similar to what the internet archive does.
Running LVM over raid 6 volumes is the 'standard' approach of many enterprise storage deployments. The 'magic' in a good RAID6 implementation is what it does when things go wrong, and lots of things do go wrong. The checksumming is great too.
At some point you can dump the sheet metal on the 'pods', just build your own rack unit. If you look at the 'big iron' systems from NetApp, EMC, and others you will see they make one big enclosure then can install simpler systems within that enclosure. What this buys you is that you can distribute enclosure services into the big box and take it off the individual boxes. That gives you better cost efficiencies. And as you point out your system can run with fewer fans, so you can put a couple of largish three phase fans in the big enclosure (dead simple and very reliable) and use them to push the air through all the other boxes. (or pull it). Then add an android tablet for the 'door' of the enclosure that tells you drive status, etc and you're practically a player in the storage game :-)
Explanation about the "GUI" comment above -> the Backblaze backup client was simultaneously written from the ground up compiling on Mac OS, Windows, and Linux. The same tree and the same source compiles on all three on EVERY SVN CHECKIN. There is one exception, which is the GUI is an extremely simple stand alone process entirely natively written to match the host OS. On Mac OS it is in Objective C in the beautiful Apple GUI layout editor, on Windows we use Visual Studio and C++ and Win32. The firm rule is these GUIs are ONLY allowed to edit one or two simple XML files, and all the real encryption, compression, transmission is done by other cross platform processes. On Linux we configure the XML file with "vi". :-) The X-Windows GUI has not even been started.
XML is a pain, but holding up release due to lack of a gui is sorta silly. Plus I imagine, like me, most multi-server users would be rolling out a standard xml config anyway without more than a simple string substitution on a per server basis.
In fact, the only GUI I have running is on my desktop.
> crontab -l
0 1 * * * sed -E -i .bak 's/net_throttle="[0-9]+"/net_throttle="100"/' /Library/Backblaze/bzdata/bzinfo.xml
0 10 * * * sed -E -i .bak 's/net_throttle="[0-9]+"/net_throttle="40"/' /Library/Backblaze/bzdata/bzinfo.xmlI've been looking for an off-site backup solution for a (nerdy and technically competent) home user for years, and yours is the only one that I could afford (poor recent BA here).
Please?! I would pay $10 a month (probably more, really) for that. Even if you don't want to support it, could you do it just for me? It'll be our little secret!
Backblaze is designed to be used with absolutely no configuration (for many users they have no idea where their Outlook.pst file is and we don't think they should have to know), and the only way we could figure out how to make this work was to backup EVERYTHING on your system unless you explicitly exclude it. So bzinfo.xml is basically a flat list of excluded folders you do not want backed up. There is also a throttle in there if you don't want Backblaze to utterly destroy your network uplink, and a few other small settings. It's pretty straightforward.
With that said, we really pride ourselves on easy to use software, so it goes against everything in our DNA to release software with NO GUI at all, but maybe we'll give that a serious thought. If you are using Linux, you probably aren't the average Mom & Pop user. :-)
Call it alpha, see how much demand there is, and the extra money you bring in might be the motivation to finish up the pretty. :)
I just want to join others asking here for a Linux version. Please do release whatever you have. And actually I would prefer a command line tool, so I can use it in my cron.
Similarly, the CrashPlan service itself is written in Java and for binary compatibility, only relies on a little binary library loaded via JNI to accommodate the differences between Linux and Solaris, not to mention Windows and Mac. It doesn't even support Nexenta (OpenSolaris kernel + Debian userland) directly out of the box, but I was able to hand-combine the bits from Linux and Solaris installs to get something working: the moral being, for OSes targeted towards more technically adept users, you may not need as much work as you think, if your app is self-contained enough.
PS: I'm already one of your costumers and I'd love to be able to use your service in my linux machines.
Recently was as low as $107 from amazon in retail package:
http://camelcamelcamel.com/Hitachi-Deskstar-CoolSpin-Interna...
However it seems in contradiction to the AWS, Rackspace model, which is a race to the bottom in that there are many competitors and they are selling a commodity (independent of the other high value services they are selling). There is some threshold of volume that is key in order to make money in that space.
Also, most businesses buy from the Dells or the HPs because they don't have the in-house expertise to manage a more bare-bones box (or more likely just don't want to). The companies that have the need/capability to manage more raw storage could just take the plans and build them out anyways I would imagine.
$2100 per month for an entire rack worth of Pods (space, power, connectivity) $74,000 is the cost to build 10 Pods to fill that rack. If the Pods are assumed to have a lifetime of 3 years (most will last longer, but lets depreciate at this rate), and if the cost of capital is 20%/year, this equates to a monthly "payment/amortization" cost of $2750. Thus leading to a total cost per rack of $4850 ~ lets say $5000.
1350TB of raw storage is provided by this rack, which can be scaled to 13/15 to account for RAID6 (as revealed by brianwski here) - thus leaving 1170TB available for use. FS overhead etc, lets take this to provide 1PB of storage.
So, essentially, storage costs backblaze about 0.5 cents ($0.005) /GB-month. There are other costs ofcourse, Sean (amongst others) needs to get paid, etc. To go by a common thumb rule, for a minimum sale price, one third of sale price should be profit, another third should be org expenses/marketing/everything else and the remaining third should be the actual cash cost of the building/providing the product/service.
So very roughly, Backblaze could provide storage at about 1.5 cents /GB-month. Factor in 3-way software-level redundancy of data, and you are now upto 5 cents/GB-months for a very high quality storage service.
Contrasting this with 15c/GB-month that Amazon charges (in addition to transfer charges), I do have to wonder why Backblaze wants to stick to the "unlimited desktop backups" business. Even Google storage charges 17c/GB-month, in addition to per-request charges.
Its quite possible I have some factors wrong here and If anyone can spot anything wrong, I'd like to know. Nevertheless, it seems from the numbers provided that backblaze could make a killing in this market. I know I would be interested in using a pure storage backend - equivalent to S3. I use tarsnap and if tarsnap could reduce its backend costs by two-thirds, I know I'd be very happy.
What am I missing?
edit: wrote PB instead of TB. Numbers remain correct, though
On that note: I could not find any real detail about this - do they encrypt your files? what kind of privacy/security do they promise? I am assuming the media files do get de-duped across all users, but what about other, personal docs, spreadsheets, pdf etc?
Tools like: http://lifehacker.com/5788550/mount-your-amazon-cloud-drive-... will let you mount the drive in windows - I have to believe it will only be a little while before Amazon lets me mount this drive with a utility they provide.
Also, considering saving money on hardware costs is a key factor in Backblaze staying competitive, they must be saving money elsewhere and/or have other competitive advantages. Otherwise releasing this information would be akin to publishing a restaurant's 'secret sauce'.
On the other hand, I read this article and I read their last one, when it too shot to the top of HN. Geeks go nuts for hardware porn, and are a great audience to sell these kinds of services to. So the marketing benefits are probably quite substantial.
As a side note -> Backblaze only does one thing -> HTTPS POST. In user space. We do NOT extend the kernel, we do not have drivers (at all), we simply read files (we don't even write to your files), compress and encrypt them in RAM, and push them through the completely common HTTPS. It is unusual for Backblaze to cause any problems except a sluggish network or a hiccup in a skype phone call.
I sent an email to sales support explaining my issue on OS X Lion when I was trialing your product, but did not get a response.
Please feel free to contact me at mqudsi@neosmart.net. What I was seeing seemed to be that the software was flooding the network card (wi-fi on a slow WiMax connection) with requests, causing it to timeout entirely such that I could not even ping my router. This doesn't require drivers, kexts, or anything and can be replicated in user mode. It's pretty much akin to DDOSing your own connection.
While I ended up purchasing a Mozy home license for a year, I do not think I'll be staying with them as I am not satisfied with either their client or their backend.
--edit--
I know it's RAID6 from the article. What I'm wondering is, how many drives in an array? How many arrays per box?
Regardless though, super impressed by the work into rolling your own hardware, hope you guys continue to do well.
We've had good luck so far with using small USB flash drives for booting big file servers. We keep the drive image pretty generic and if there's a problem with one, we just replace it with a cloned USB flash drive and reboot, no problem.
It doesn't seem to hurt performance at all for these kinds of uses -- although we do set it up without swap to keep the life of the USB flash device reasonable, which might or might not work in your case.
The ECC RAM absolutely does find and corrects problems (we see them in the logs). However, just to be absolutely clear we would not need ECC RAM -> Backblaze checksums EVERYTHING on an end-to-end basis (mostly we use SHA-1). This is so important I cannot stress this highly enough, each and every file and portion of file we store has our own checksum on the end, and we use this all over the place. For example, we pass over the data every week or so reading it, recalculating the checksums, and if a single bit has been thrown we heal it up either from our own copies of the data or ask the client to re-transmit that file or part of that file.
At the large amount of data we store, our checksums catch errors at EVERY level - RAM, hard drive, network transmission, everywhere. I suppose consumers just do not notice when a single bit in one of their JPEG photos has been flipped -> one pixel gets every so slightly more red or something. Only one photo changes out of their collection of thousands. But at our crazy numbers of files stored we see it (and fix it) daily.
But the Supermicro board says it can: http://www.supermicro.com/xeon_3400/Motherboard/X8SIL.cfm
The board also supports Xeon processors, so it might be that it only supports ECC with Xeon. However, I didn't see anything in the docs to support this, instead it says "Dual Core processors of Ci3 and Pentium: support ECC UDIMM only"
I'm curious since I ended up springing for Xeon in my system specifically for ECC. Now I'm wondering if I made a mistake...
"The X8SIL/X8SIL-F/X8SIL-V supports up to 16GB of DDR3 ECC UDIMM or up to 32GB of ECC DDR3 RDIMM (1333/1066/800 MHz in 4 DIMM slots.)"
Below are what we assume are ECC errors from an i3 based Backblaze pod, when this happens it is usually bad RAM (bad enough to crash the pods, replace with fresh RAM repairs it):
### IPMI LOG: ####
ipmitool sel elist | grep -v "Fan FAN" 69 | 01/04/2011 | 20:21:52 | Memory | Correctable ECC | Asserted 73 | 01/04/2011 | 21:15:50 | Memory | Correctable ECC | Asserted
Have you tried using ZFS on one of these?
ANOTHER option down this line of thinking is switching to btrfs, but we haven't played with it yet.
By the way, at Backblaze I've felt like we have had to implement several things that I would have guessed would be standard "off the shelf". One example: When a customer wants to download a restore file with a web browser, are you aware there are no checksums for over-the-network transfers other than the built in (completely unacceptable) 16-bit TCP checksum? You are virtually guaranteed to have an undetected corruption within 30 Gbytes of download, which is basically what we like to call "a totally average customer restore". So Backblaze had to write our own custom reliable, restartable "downloader". It boggles my mind that the whole internet is throwing undetected errors on HTTP downloads and nobody cares to fix the protocol?!! Where the heck is Google, Apple, Facebook, Microsoft, or <insert standards body> defining a standard for web browser downloads larger than a few GBytes?
The last time we tried Linux for this, it couldn't do it. Maybe that's gotten better though.
We also tried DragonflyBSD, but it had hardware support issues, and OpenBSD, but unfortunately OpenBSD just simply cannot do large filesystems. At all.
ZFS-on-FreeBSD is the way to go at the moment, I think.
I actually use ZFS on Mac OS, as well, and have been happy with it there (thanks Dustin Sallings!).
As for HTTP, I agree with you entirely! There is actually a standard that lets you put a hash (SHA-1 or SHA-256 or something) on an http anchor link, and the browser will verify it when the download finishes. It hasn't really gained very wide adoption though. Personally, I think something akin to Bittorrent is a better solution, since it doesn't have to redownload the entire file when it detects an error. It's ironic that our videos often have better data integrity than the graphics driver installer that I just downloaded, or the browser that I downloaded yesterday.
Two possible ways of solving this: make a download into e.g. 16 MB chunks and append a SHA-1 checksum for each chunk (not enterely unlike BitTorrent). Then re-download the chunks where the SHA-1 doesn't match. Another solution would be to use e.g. Red Solomon error correction.
Microsoft does the same thing -> they have a custom application to download their OS updates reliably. They do not use Internet Explorer's regular download capability because of the limitations.
It baffles me that the big browser companies (Google with Chrome, Apple with Safari, Microsoft with IE) are not interested in building a standard reliable downloader. I mean, what else does a web browser really do?
I'm not sure which HTTP client implementations reliably test those checksums, though. (Or whether CRC32/Adler32 would be adequate for 30 gigs!)
But brianwski's point remains that neither HTTP encodings, nor HTTPS can detect certain errors like in-memory corruption of the data after it has been read from disk, but before it has been sent out over the HTTP connection.