Facebook's Top Open Data Problems
research.facebook.com
research.facebook.com
EricBurnett was talking about appreciating (some of) FB's values, not relishing the scale of their challenges (another reason people join FB). It's the latter reason that I imagine engineers joined The Manhattan Project (other than those who viewed it as a way to protect against nefarious forces in the world, valid or not).
My experience was really similar to my Google internship, and probably even closer to a "cool startup". I know several people who worked on Google, Facebook, and X (with X another major Silicon Valley company) and say that the first two were a lot closer to each other than to X.
In terms of the Facebook product one cannot deny the obvious: They touched THE nerve on the internet. World wide. Across languages and cultures.
I don't like the embodiment of the product at all. At the very least it has usability and privacy problems. Yet more people use it successfully than any other web app in the world. So, what do I know? What do experts know?
A similar thing could be said about CraigsList. It's 2014. Every time I use the site I cannot believe what I am looking at. Yet I and lots of other people keep using it. It works.
Facebook has a general lack of elegance (whatever that means). The kind that happens when a product is thrown together and evolved over time. Evolving anything over time means the output "naturally selects" (subverting the theory here) to the environment created by it's users.
They survive because they optimized for what is important to their users. Grandma couldn't care less about UI issues or searchability. She wants to see her grandchildren's pictures and videos. And for that it works very well for a huge percentage of the planet.
At some point it becomes almost impossible to break the mold and clean-up what might be less than ideal. Why would you? It works. Another "Innovator's Dilemma" [1] situation to a large extent.
[1] http://www.amazon.com/The-Innovators-Dilemma-Revolutionary-B...
https://github.com/memcached/memcached/blob/master/memcached...
is more like 4000 lines of C (by guesstimating the amount of comments etc).
4000 isn't that much either & whats memchached? A hashmap?
He maybe means 300 lines of relevant code (+ ~3700 sugar)
I agree that 4k lines isn't that much, but it's an order of magnitude off from 300. And again, that's just for that one single file.
I was reading somewhere sometime back that each user at fb has its own database. I think that is not possible.
edit: I am googling now again on this topic. First link found is http://www.quora.com/What-is-Facebooks-database-schema
The majority of core information (attributes of people and places and pages and so forth, as well as posts and comments) is stored in MySQL and queried through TAO.
Some data is primary stored in things like HBase, such as messages.
Non-primary-storage data (indexes and so forth) exist in various forms optimised for different workloads - so data in either MySQL or HBase might also exist in Hive for data warehouse queries, or in Unicorn for really fast search-style queries.
Other data (such as logs) might reside in one or more of the various data stores, such as Scuba, Hive, HBase, and accessible via Presto, for example.
TAO: https://www.facebook.com/publications/507347362668177/
Unicorn: https://www.facebook.com/publications/219621248185635
Hive: https://www.facebook.com/publications/374595109278618/
Scuba: https://www.facebook.com/publications/148418812023978/
Am I being too cynical in thinking that Facebook is intentionally misleading its users in an attempt to bump up their metrics? It interests me that they are seeing jumps in their mobile users (and consequently, ad sales) at the same time that I have been receiving more notifications than ever. Interestingly, the slowdown in fake toast notifications coincided with their quarterly earnings report that show mobile ads accounting for an increasingly large portion of revenue and also mentions an increase in mobile user usage.
Comparing Q1 with Q2 with Q3, Q2-Q3 showed double the increase in ad revenue percent from mobile (59% to 62% to 66%). Maybe this is just all anecdotal evidence, but it seems like these sort of fake notifications should either not be sent out (failure of the system that keeps track of what user receives what toasts) or there was a conscious effort to send these notifications....
Interestingly, Twitter's metrics don't appear to show any similar rise in the rate of adoption.
http://developer.android.com/guide/topics/ui/notifiers/toast...
So, how much does the 4PB of data go down to when it goes into hive? My guess would be something like 200TB.
Is it zipped (or lz4/lzo/zopfli/lzma whatever)? Or is it just "distilled"?
Most of it likely gets tossed into a program to determine if somebody actually needs to do anything, or if something is actually breaking.
For example raw user interaction doesn't really grow. While the event is likely ~1kb or so of raw data, at the end your just incrementing a 64bit counter.
This is a baseless exaggerated post but should shed some light on.
Well, as a non-facebook user, I think 99.9999% of facebook data is useless :) But facebook is in the habit of tracking everyone's surfing habits across the web (through "like" links), and I assume they do more than just "increment a counter with it", even if they don't keep every single detail.
This is a problem I'm actively trying to solve for a project, so if somebody knows the answer, please get in touch!
EDIT: More specifically, how do you efficiently cache your data such that an assoc_range query can be answered from cached without O(n) operations on your application servers. Memcached can't answer a query like "give me 50 items starting from position 0 in the list" as far as i'm aware, so you'd need to pull the whole list to the application server and slice it up there. When you consider that you want the 50 most recent items too, maintaining sorted lists adds extra complexity.
Application -> TAO -> Cache -> Database
Lets say my page size is 50 and I want to view 3 pages of comments for the photo from my application. My application makes the following TAO queries:
assoc_range(P1, 0, 50)
assoc_range(P1, 50, 50)
assoc_range(P1, 100, 50)
A naive implementation might be to store the list of all edges for P1 with a key of "P1". To answer te above 3 queries, TAO then needs to pull "P1" (all 125 edges) from memcached 3 times to answer each of those 3 queries and slice the edge list up on the TAO application server... Not great, but probably an improvement over hitting the DB for it (up to a certain list length at least).
A less naive implementation might be to store the edges in buckets of 50, such that the 125 edges are stored with keys of "P1_0_50", "P1_51_100", "P1_101_150", but then time ordering comes in to play...
If my application now wants the 50 most recent items, we could store the edge lists by created date descending and I can retrieve "P1_0_50" from the cache and guarantee I have the 50 most recent items. However, lets say 10 new comments are posted... Now I need to update all my cache pages to ensure the ordering is correct, which is horrendously ineffecient!
To fix this issue, edge lists could be stored in created date ascending order instead, but then how do I know which cache page to fetch to retrieve the 50 most recent comments (seeing as "P1_0_50" is now the oldest 50 items)?
I hope that makes sense!
So, when you add a comment, TAO updates its internal structures with the new comment in the right place (after the DB is updated), and there are no "keys" that need to be updated beyond that.
I love to try build a golang version of TAO as good mental exercise.
Anyone if something similar in golang or any other open source packages exist already?
Anyone else interested in such thing?
A lot of that data is also not tied to individuals either - for example the access logs for the CDN (which, being on a different domain by design, does not share cookies so is not attached to an account) even reasonably heavily sampled is probably tens of gigabytes a day, and is rolled up into efficient forms for queries in various ways. A lot of it isn't even about requests coming through the web site/API - it may just be internal inter-service request information, or inter-datacenter flow analysis, or per-machine service metrics ("Oh, look, process A on machines B through E went from 2GB resident to 24GB in 30 seconds a few seconds before the problem manifested").
(Not that it makes too much of a difference at this scale, but it is closer to 860M daily actives.)
I wonder if they can predict with some percentage accuracy on what any particular active US user might vote for today base on the user's graph data?
So 4 PB per day, but only 300 PB total?
Balderdash. memcached is actively hostile to modern infrastructure and it's not being actively developed except for routine maintenance.
The first time someone stores an entire data structure in a memcache key is the moment you've lost. Playing "read blob, deserialize, update, serialize, write blob" is just dumb when you can avoid it.
Other moments of great failure with memcached: needing to implement client-side or proxy-based replication, having the memcached process refuse to cache more things because a slabs become broken, the first time your developers don't understand memcached LRU is per-size category and not global, ....
Just Say No to Memcached.
Why gild the lily?
> needing to implement client-side or proxy-based replication
It's best practice to replicate the persistent store for availability, not the cache.
Almost all your other concerns appear to arise out of fundamental misuse. It's a cache, not a panacea to cover up basic design flaws in a primary data store.
Mcrouter was the first piece of software I worked on at facebook. It's nice to see that you like it! (Though the only thing that lives on from me is the umbrella protocol)