LinkedIn's Data Infrastructure

I was similarly surprised to read "three people work on [Facebook] photos, the largest photo site on the internet." (http://ow.ly/2k7ka)

My theory is that they have good managers that take care of all the BS and free their minds to get tons of work done :-\

The article sounded like they just meant that 2 engineers work on the "people you may know" thing. I'm pretty sure there's more than that on the SNA team. (www.sna-projects.com)

If you have top-notch engineers, it takes very few of them to manage very complex problems. At that level there is also an advantage to having few people: it reduces the communication and coordination overhead. Assuming the general rule that a great engineer is 100x as productive as a typical engineer, reducing overhead is 100x as effective as well.

It doesn't work this way for all situations. In the case being described here, the data is huge and the algorithms are complex, but it sounds like the software itself is probably rather small. That makes it suitable for a small team of very good engineers. In situations where the software is huge but none of it is particularly complex, you're better off with a large team of average engineers. (Of course, if you put very good engineers into that position, they will probably re-factor your software down to something more manageable.)

Pagerank is a like a markov chain - you iterate the same dataset over and over until you're happy with the result. If 20 is good enough, its good enough. Good explanation here: http://www.iterativemapreduce.org/samples.html#Pagerank

Whereas if you include signals from multiple sources, the joins are each one MR job, never mind the calculations.