682 karma · joined June 7, 2012
Amazon 1997-2004 (supply chain, product catalog, website engine, service frameworks, image processing and serving)
Apple 2005-2006 (product catalog, image rendering)
Facebook 2006-2015, 2020-2021 (storage, networking, incident management, site performance & investigation, internal & external threats)
Maybe I gave HBO too much credit. Still, I do think Netflix has a bit of a Craigslist/Amazon issue where so many people are used to it working a certain way that it's now hard to change the UI in material ways without causing a drop in revenue. I've used their new web UI, and it's a little better, and I'm glad they dropped their red-heavy color scheme, but it's still basically the same clickable linear lists of shows arranged into a series of rows. I guess I'm just cranky that there aren't more visually interesting ways to discover new shows--I find that my eyes glaze over after a minute or two of browsing with these kinds of UIs, and everyone seems to use them.
I think the worst UI award goes to iTunes though. It is so hard to discover new shows and movies on there. I can find things I already know about, but when I'm searching and trying to find something new and fun, it's hard to find cool stuff. I basically click on related shows for something I already like until I find something new, but it's not fun and I find better recommendations on Netflix.
1. The shift from cheaply produced, low quality content, to higher quality "must see" shows. It is definitely the must see shows that have drawn me to HBO and Netflix. I don't just randomly watch whatever's on HBO--I have it for Game of Thrones, Silicon Valley, etc.
My use of Netflix is a little more dual-purpose though; there's certainly Netflix-produced shows that I like a lot, but I often find myself nostalgically surfing Netflix to watch old shows like ST:TNG, or to watch favorite movies again. Even though HBO and the different TV providers offer on-demand viewing, there's something about their offerings that makes me much less likely to browse their libraries and watch random content. The paradox is that Netflix's UI is on average worse than theirs, so my preference of browsing shows on Netflix is not a usability thing, it's an emotional thing.
2. The comments about how Netflix has commoditized time since we no longer have to watch shows at a certain time each week. And how this is borne out by their strategy of releasing all episodes at once for shows they produce. I've found, like the author, that releasing them all at once reduces some of the fun and anticipation of watching a new episode and talking about it with your friends. I've also noticed that I tend to binge watch shows when new seasons drop, and tend to fast forward through less exciting parts of the episodes a lot more than normal, since I feel I have to get through the whole season so I can talk about it with my friends. I get really impatient by the time I'm 80% through the season. But at least this shows how aggressively they are committed to giving people their time back and letting them consume the content on their own schedule.
3. I had no idea HBO was producing so many projects at once. Hollywood Reporter said they are working on over 100 projects right now. I didn't consider that Netflix and Amazon could position themselves not just as alternatives to people who couldn't get a seat with HBO, but could also potentially produce the show and get it online faster (e.g. steal good projects from HBO with the promise of fast results, not just taking their rejects). I still think HBO will on average produce higher quality shows for quite some time since they have so much practice at it, but it sounds like they are spread very thin. I would love to read some analysis on how HBO produces their shows and what their org structure is like there.
BTW, you can store private keys in a HSM on your private key server and that's an additional layer of protection, but nobody uses TLS accelerator cards anymore that I'm aware of. You can simply do the relevant crypto on commodity CPUs and still have plenty of throughput.
1. Store your private keys in a separate process, either on the same machine or a remote, more secure host. Offload private key operations to the private key service. This obviously requires an encrypted connection between your webserver and the private key service, but you get good gains in security from no longer having the private key in your public facing web server's address space.
2. Another technique you can add is to separate the data plane from the control plane on your public facing webserver--that way you have a stripped-down process that is just handling the low level reading and writing of buffers to the wire, and have a fast pipe connecting it to a separate process doing all the HTTP logic. That way, you can lock down the data plane service and make it harder to exploit since it has a much smaller surface area than a full fledged web server.
It does always make me think twice before committing to using a startup's product, you never know who is going to fail and sell assets to the highest bidder, or who is going to get acquired by a big company whose ideals you don't like.
The problem is that oftentimes, the data is one of a company's most valuable assets, so you can't just make it off-limits since that would erase a lot of the value. I just wish there was a better legal framework to handle these kinds of situations.
I dig the other stuff that Dragonfly is doing too. I've been a longtime BSD fan and I like that there is a strong-willed counterpoint to FreeBSD with a different architectural philosophy. The pressure pushes FreeBSD forward faster than if Dragonfly wasn't around.
If you can get the smear to be gentle enough to not affect your apps, it's a nice workaround for the general problem. There's just so many components running inside a modern infrastructure, and it's hard to get them all to behave correctly during a traditional 61-second-minute leap second transition.
And that 2012 leap second linux kernel bug was pretty epic. AFAIK, people who are smearing didn't experience it.
http://googleblog.blogspot.com/2011/09/time-technology-and-l...
I just commented because I wanted people to know that it wasn't like there was no software focus before Joe left--things weren't as binary as that. The software focus shift had been in progress for some time before he stepped down.
And yeah, feels good to talk about FB with no filter. :-)
I'm excited to see what Alex does since his credentials are stronger on the technology front, but the team's shift to a technology focus has been happening for a long time. I view this as another hugely positive step in that direction.
It took Proxygen less than 18 months to go from inception at a hackathon to running all of Facebook's production HTTP traffic. And we could have shaved that by a few months if we had made smarter decisions along the way.
DNSSEC was never something we were aiming for, so it's fine that TinyDNS doesn't support it out of the box.
There have been detailed talks about Proxygen's architecture at various open houses in Seattle and Menlo Park, and they should have HTTP2 support working well soon. I'd expect there's going to be more detailed blog posts on Proxygen at some point. Or you can check out the source code too.
Fragmentation isn't really an issue for either Shiv or Proxygen. I don't think they will share the rate but it is very low.
There were a few things that were interesting or novel about Haystack. CDNs were only giving us a 80% hit rate on photos, and even back in 2007 when we first starting working on Haystack, that was an immense number of misses to serve from the origin. That heavily informed the design goals of Haystack, which were to have a very I/O efficient userland filesystem that used direct I/O to squeeze as much iops from the underlying drives as possible. We also wanted zero indirect I/O, so we used an efficient in-memory index so that all seeks on disk were to service production reads versus indirect index blocks or other metadata. Lastly, Haystack is very very simple. We made it as simple as we could and eschewed anything that looked too clever. I actually think its simplicity is one of the things that made it Hard. It would have been much easier to design something more complex.
If you have any questions about the Haystack paper after reading it, I'd be happy to answer what I can. Just send me an email.
Regarding the datacenters, you're kind of simplifying it, but that's okay. There's a lot more orchestration, automation, and integration, and I wish I could give you a tour of one of our OCP datacenters so you can see it firsthand. There's plenty of things I wish I could talk about on the datacenter front that I feel are truly amazing and without precedent, but I can't until they are public. There's an excellent chance that they will become public and likely even part of OCP in the future though. I'm really proud of how much of Facebook's datacenter architecture has been shared with the world for free. It gives me warm fuzzies all the time.
Anyways, thanks for the fun thread, and let me know if you ever want to talk storage sometime.
First, the abstract thing. I don’t know if most engineers outside Facebook appreciate how much activity there is on the site each day. Hundreds of billions of likes and comments. Billions of photos uploaded. Trillions of photos consumed. And growing each day.
And this is for an “online application” in the sense that the data is live and constantly changing. We’re not talking about crawling the web, storing it, doing offline processing, and then building a bunch of indices (which is a different but still legit kind of hard). This is an immense amount of live data producing an even more immense stream of live events; trillions and trillions and trillions that need processing, live and on the fly, every day. It is hard to underestimate how difficult it was to build an application and backend that could drive this kind of social platform. In terms of liveness and scale, there really is nothing out there that can touch Facebook, by orders of magnitude. That is a solved Hard problem in my opinion, if not a meta one. But in my opinion, the largest one.
Here’s two concrete things that I thought are good examples of some Hard problems that Facebook has solved:
1. A global media storage platform that each day is capable of ingesting billions of new photos and videos and delivering trillions as well. This includes Haystack, F4, and Cold Storage, which store hot/warm/cold media objects for Facebook. Each storage layer has specialized, custom software running on distinct storage hardware designed to take advantage of the requirements for each layer. Facebook is the largest photo sharing site in the world by orders of magnitude, and they had to build a very custom photos backend to handle the immense load. I’m not even mentioning their terabit class Edge platform which has a global constellation of POPs that accelerates application traffic and caches popular content close to users. Facebook’s global media storage and delivery platform is truly a unique asset.
2. A datacenter architecture focused on high power efficiency and flexible workloads, comprised of custom built: datacenters, racks, servers, storage nodes, network fabric, rack switches, and aggregation switches. Plus some other things that haven’t been made public yet. This architecture let application developers stop working around various physical performance bottlenecks with compute, storage, cache, and network, and instead just focus on optimizing what was best for the application. At the same time, this new architecture also greatly reduced infrastructure costs.
There are other solved Hard problems I can talk about that are public, and others that I wish I could talk about that aren’t yet. However, I’m not trying to exhaustively defend what Facebook has built, I more just wanted to respond and show a few examples of the very special scale Facebook has and some of the extremely hard problems they’ve had to solve.
Facebook actually built many major pieces of its infrastructure itself after finding no suitable solution in open source or commercial vendors. Efforts like this were happening even back in the early days of FB infra. I could list out many publicly known Facebook infrastructure projects that solved Hard Problems. And not just in recent years either. :-)
Haystack still handles the initial writes and the burst of heavy read traffic when an object is fresh. It has a higher space overhead of 3.6x because it's optimized for throughput and latency versus cost savings. The Haystack tier's size is now dwarfed by the size of the F4 and Cold Storage tiers, but it still handles almost all of the reads for user blobs on Facebook due to the age/demand exponential curve.
After a haystack volume is locked and has cooled down enough, it moves to F4 to bump its replication factor way down for cheaper, longer term online storage. And then Cold Storage is used for the older stuff that gets barely any reads but that you still want to store online for perpetuity.
That is why the team that works on media storage is called Everstore; they take storing people's photos and videos seriously and view it as a promise to keep your memories available forever. It feels really good to see photos from 5+ years ago and have them still work, and someday there will be 30 year old photos and videos on Everstore as well.
Source: I built Haystack with a couple other people and founded the Everstore team. :-)