Why WhatsApp Only Needs 50 Engineers for Its 900M Users

Yes yes, I do believe there's some boosterism going on when their success is entirely attributed to Erlang/FreeBSD alone. Sure the stability and scalability of the backend may have been helped by using Erlang but the fact that they got to scale at all has to do with all these other business/cultural/timing/design factors that I mentioned above.

Btw, the one factor I forgot to mention above was the call to use phone numbers as the namespace for user identity. I believe they were one of the early ones, if not the first ones, who eschewed email and/or social login and went straight to phone number. That made them a direct replacement for SMS in large parts of the world where SMS costs were meaningfully expensive to the population. One more example of focusing outside the US market in general and outside of Valley bubble in particular.

The servers, on the other hand, have to serve all the million users connected at any single moment. That requires a lot of parallelism, and that's the most likely place where Erlang was successful.

What other apps besides social networking and communication apps even get to hundreds of millions of user?

You are quite the optimist.

> I feel the language choice is probably the least contributing factor in terms of what's interesting at that scale.

However, as they say "right from the horses' mouths", WhatsApp engineers have said that Erlang and FreeBSD are critical tools for their success. One can accuse them of lying, of course, but why?

Here is an interview with Eugene Fooksman:

https://pdincau.wordpress.com/2013/03/27/an-interview-with-e....

---

...and the consensus in our team is that it is largely because of Erlang. We’re managing to serve huge amount of connections from single front-end server...

---

There is the "That A Billion With A 'B'" talk by Rick Reed at Erlang Factory (already posted somewhere) in here.

Here is another older directly from their website:

https://blog.whatsapp.com/196/1-million-is-so-2011

Clearly states Erlang and FreeBSD are important.

> These things just happen and have to be worked through no matter what system you are using.

Yeah but that is a bit too generalizing. It is like saying "well the language is Turing Complete" argument, yet it is but one tool is much better than another. Surely Twitter runs on JVM, Google doesn't use Erlang etc etc. So you can do it many other ways. And, at least in this case, they certainly picked the right tools, as we can see from their success. The number of users * amount of messages processed / # of engineers is a not a bad metric to see how good a tool is. And Erlang is a great tool here.

(Another example is when you use the smartphone to access the Internet, chances are 50% of the time that connection is set up by Erlang. It is there in the background doing its job, not crashing, setting up even an order of magnitude larger amount of data than we see from WhatsApp).

It's here https://www.reddit.com/r/programming/comments/3l2spe/why_wha...

In its entirety:

I can explain a bit more in depth than the author did.

Erlang uses a model known as the actor model. This means that each part of your program is a completely independent entity like a mailbox, known as an actor. For one actor to communicate with another actor, it simply puts a message in the intended actor's mailbox. Since each actor is independent, this means that we can spawn as many actors as we want to read messages from a single mailbox, and do what is requested.

In terms of what WhatsApp is doing, this means that they can have (potentially infinite) conversations going on between users all at the same time. If any part of their pipeline is working too slowly (there are more messages being put in the mailbox than taken out) they can just spawn additional actors to read mail (at any time they want, even programmatically). Additionally, as alluded to by the author, in Erlang you can do something known as "hot loading," which means spawning newer versions of actors (when the implementation changes) while still running the old actors so that there is never any loss of service.

Haskell is similar to Erlang in its ability to scale. In fact, there is a library in Haskell known as HdpH-RS that enables fault-tolerant (the programmer does not have to worry about intermediate failure at all) distributed computing/memory (tested up to 1400 cores). The main reason for this is that Haskell is strictly-typed, meaning all types must be defined (or derivable) at compile time, and immutable, meaning that variables cannot change their values after they are defined.

Due to all of this strictness, you never have to worry about errors found in languages like C such as segfaults, or improper use of a function since you can enforce usage with types. For instance a function that should only accept unit vectors would use a type UnitVector instead of Vector (Note that this approach is generally a bad idea in C/C++/Java etc). Essentially, if your program compiles, then it will run correctly (generally the only real errors you run into are logical ones -- the algorithm itself being incorrect). This makes it really easy to refactor code later on. The reason Facebook can handle urgent tasks so quickly is mostly due to Haskell's generics, which are like templates from C++ or generics in Java. The main difference however is that Haskell lets you create generic mixins for types known as type classes. This means that you can write extremely general code to handle almost anything in the high level. Then, whenever you need to handle something new, you just have to choose which mixins you want and just add them to your type by defining the interface. For example, I could make a Multipliable type class, I would require the programmer to only define the multiplication operator for his type, and then it would automatically create things like pow or sqr.

Erlang and BEAM were designed for highly concurrent messaging applications. Surely that's of some importance.

They could be using really good tools. Or each of those 50 programmers are 10x more productive than regular programmer (I don't believe in this imo).

Plus, if you are going to hire in this fashion, why even bother with resumes, or extensive resumes? Just bring people in for a test? If they do well--hire them?

I always felt a test is the fairest way to hire new employees. Tests do away with upperclass privilege(better contacts, padded resumes, etc.) I wish more companies vetted employees in this fashion. It might get rid of something I have always dreaded, and get slightly nauseous when I see the behavior in action--and that is Networking. Sorry, I always thought it was phoney.

I still imagine your company relies heavily on resumes, and uses the test--because they can? Meaning--there's still a lot of desperate software engineers out there; willing to do anything to land a job?

You example, unity, isn't a language, but instead an off the shelf engine and toolchain.

But just like the development on an individual game seem less impressive technologically the more they rely on existing tech [1], admitting "Yeah, we forked ejabberd and rigidly fight scope creep" doesn't have the same ring as hyping up features in the runtime of a hip yet obscure programming language.

[1] Fun side note: If you watch the games press interviews devs, pay attention to when they ask about what engine they use. The press don't and viewers often don't know anything about software projects, so the devs are reluctant to to admit how much they've relied on off the self components for their admittedly-huge undertaking. Fun to watch the paid hype men try to talk around that.

I haven't heard of anyone who has figured out how to measure an engineer's abilities.

Some people test algorithms. Others test how often they've shipped things. Most just look for years of experience. A few even test whether you know certain facts.

Who doesn't say that though? No company says they hire average engineers.

But I agree that their success does kind of prove that they succeeded in that.

If you're interested, my contact details are in my profile. Happy to have a call/Skype. It's an important topic.

[1] http://fortune.com/2013/11/27/by-the-numbers-how-foxconn-chu...

You can’t get electronic tax filing without technology, you can’t get full open data availability with nice diagrams like https://www.destatis.de/bevoelkerungspyramide/ without lots of IT workers in the government (this is available for every data collected by the German Federal Statistics Agency, btw), and you can’t get stuff like http://ims.kiel.de/extern/kielmaps/ which show even lanes and zoning for cities without people working on it.

Part of the job of governments is also always having reliable, high quality data about their country, for government purposes. But it also means they need to publish those for citizen, and they need people taking care of that.

Whitelisting should always be an option - it's simple to achieve technically and allows the consumer to have complete control.

Edit: And the fact there is no whitelisting is also the reason I removed the app from my phone.

Obviously, creating Whatsapp is nothing near a weekend project. But I think there founders should be commended especially for avoiding some classic mistakes: the NIH syndrome and going on a hiring spree once the investments started rolling in. Their managerial skills are at least as good as their engineering skills.

Do you? If yes, please enlighten us. Otherwise I don't see the point of your objection. No one said that it can be done in a weekend. All that the parent comment said is that the fundamentals of building WhatsApp are already solved. He/she didn't dismiss it. Just because something isn't rocket science doesn't mean it's not useful.

As for Facebook, people used to dismiss it not because it was seemingly too easy to build but because they failed to see its usage. Long before it became a platform it was pretty much a waste of time. Obviously anything with more than a hundred MM users is difficult to maintain.

What I in _no_ way could do is rewrite fb or twitter at the scale these systems operate today with a billion users and even more posts and data points. I don't agree with the sentiment of the previous poster, but fb & twitter were initially basic CRUD apps.

There's a ton of work involved in making a platform like WhatsApp a polished user experience. There's then a ton more work involved in making it _solid_. In choosing Erlang, they chose a platform that trivialises a large part (though not all) of the latter, allowing them to focus on the former.

Is this a simple 'solved' problem that anyone could do? No. But is it a super-human feet that only a handful of people could achieve? IMO, no as well. I think people in the latter extreme like to believe this as it's something they can aspire to - to say otherwise is to devalue their vocation. In the former camp, I think it's people pushing back against this too far in the opposite direction.

Maybe the backend. I couldn't write a Blackberry Java app though which was WhatsApp's initial value proposition.

From a technical standpoint, WhatsApp, the product, and the scale they operate at are nothing new.

Don't forget WhatsApp has applications on all platforms.

Also, let's not forget to give credit to Carl Hewitt who came up with actors in the first place [2].

[1] http://www.infoq.com/interviews/johnson-armstrong-oop

[2] https://www.youtube.com/watch?v=7erJ1DV_Tlo

http://xkcd.com/1425/

Talking about ejabberd is admitting a core piece of the product is standing on the solders of giants. Something startups are often loathe to do.

Also, an army can only send so many messengers before you run out of soldiers (or operational readiness). Packets are free and plentiful. If you're on a network link so bad that you can't even occasionally receive a TCP ack, then you're very much an outlier.

One aspect of this trend of resentment you describe is that having comparatively less robust and using less efficient languages (like js with nodejs) in the servers would make them require more cloud resources.

We as humans tend to rationalize our past choices, probably a fair amount of the dismissal is from developers who think their "secret sauce" is using AWS with auto-scaling.

And the only one who wins something from this trend is Jeff Bezos and other cloud shareholders.

1. No web interface, I'm forced to use a tiny mobile keyboard

2. Identity tied to my phone number which changes every month

3. Ugly (subjective)

4. My friends are already on Facebook, why have another service that does the same thing but worse.

5. No chat bubbles.

6. No meme search

I could go on and on. They just got lucky and hit the lotto with foreigners who couldn't afford decent phones and are now locked in to their platform.

Edit: formatting

I have found other mentions about it, perhaps on Erlang mail lists, here is an interview with Eugene Fooksman:

https://pdincau.wordpress.com/2013/03/27/an-interview-with-e...

"""

And the consensus in our team is that it is largely because of Erlang. We’re managing to serve huge amount of connections from single front-end server (this is from last year’s Erlang

"""

I don't know how much clearer can it get.

Here is another directly from their website:

https://blog.whatsapp.com/196/1-million-is-so-2011

It is a bit older they clearly indicate that FreeBSD and Erlang are key and critical elements that let them operate they way they do.

This is kind of true. But there is a nice community around erlang. I always found good, valueable input at erlangcentral.org and stackoverflow.

"A method of programming where you can safely allow many different, but related computations to occur simultaneously to arrive at a desired result. Thanks to the "safety" aspect, these computations can be automatically processed in parallel (at once) by many thousands of computers to increase the speed of the program, but no collisions of data processing will occur. In other words, no two parallel processes will work at cross purposes as a result of parallelization - when the code is written correctly and concurrently."

It does, can personally confirm.

When architected correctly, the actual amount of code to be written and maintained is quite low. And an easy way to jump-start that architecture is to have other people who have amortized the cost into a service model do that work for you.