Official Google Blog: Sorting 1PB with MapReduce
googleblog.blogspot.com
googleblog.blogspot.com
(68sec * 1024) / 60 / 60 / 4 --- (68 sec for 1TB) (1024 to make 1TB) /60/60 (hours) and /4 as there are 4x as many computers....
So ideally taking the 1TB sort to 1PB on 4x the hardware would be 4.83hrs.
Google's 1PB sort was in 6 hours - So that's fairly linear. Impressive.
I was assuming that the sort was actually a partial sort - returning the top "x" (admittedly - even then isn't linear). However, I was also assuming (n << x), which would almost certainly be the case with a large data set. It's a fairly common approach, but not sure why I automatically assumed it.
If the sort is complete, then yeah, that changes things.
I'm a bit at lost here. Does that mean that one in ~50K hard disks fail within six hours of usage?
With x% annual failure rate, are the failures uniformly distributed over time? I would expect that it's probably a normal distribution over time. (i.e., no failures to very few failures in the beginning, and the failures keep increasing with time).
But the fact that the article specifically mentions about drive failures and that they use some of RAID to cope with this makes me wonder if they were using old disks. It's probably easier to get all new drives and use them isn't it? On second thoughts though, it might be much easier to tell GFS to make an extra mirror than to make sure that all disks are brand new. So, i buy your line of thinking :)
I'm actually pretty happy that we (Yahoo!) are talking about what we are doing in the Open and contributing in an Open project.
You could reproduce what we did. Obviously you can't reproduce what Google did. What's more useful?