HNHacker News
TopNewBestAskShowJobs

nocarrier

682 karma · joined June 7, 2012

IEG 1996-1997 (high traffic, image processing, CGI, SRE stuff)

Amazon 1997-2004 (supply chain, product catalog, website engine, service frameworks, image processing and serving)

Apple 2005-2006 (product catalog, image rendering)

Facebook 2006-2015, 2020-2021 (storage, networking, incident management, site performance & investigation, internal & external threats)

submissionscomments
nocarrier··on Why Snabb?
I think Luke's work on Snabb is impressive and it's cool to see efforts to make a marketplace for Snabb based solutions. I don't know if it will work out since companies can be fickle when choosing solutions, but I'm rooting for Snabb nonetheless. It's great to see people building non-trivial network things with kernel bypass.
nocarrier··on The View from the Front Seat of the Google Self-Driving Car, Chapter 2
I think people focus too much on Uber replacing consumer transportation. I think they're going to branch out into transport of goods and eat into the trucking industry in a major way, in addition to whatever they do with consumer transportation. A lot of the commoditization concerns that you'd have with transportation don't apply as much to transport.
nocarrier··on Netflix and the Conservation of Attractive Profits
You're right, Netflix does a great job of tracking state and knowing where you are in an episode. And their streaming is pretty solid too.

Maybe I gave HBO too much credit. Still, I do think Netflix has a bit of a Craigslist/Amazon issue where so many people are used to it working a certain way that it's now hard to change the UI in material ways without causing a drop in revenue. I've used their new web UI, and it's a little better, and I'm glad they dropped their red-heavy color scheme, but it's still basically the same clickable linear lists of shows arranged into a series of rows. I guess I'm just cranky that there aren't more visually interesting ways to discover new shows--I find that my eyes glaze over after a minute or two of browsing with these kinds of UIs, and everyone seems to use them.

I think the worst UI award goes to iTunes though. It is so hard to discover new shows and movies on there. I can find things I already know about, but when I'm searching and trying to find something new and fun, it's hard to find cool stuff. I basically click on related shows for something I already like until I find something new, but it's not fun and I find better recommendations on Netflix.

nocarrier··on Netflix and the Conservation of Attractive Profits
This was a fun read. Stuff I found interesting:

1. The shift from cheaply produced, low quality content, to higher quality "must see" shows. It is definitely the must see shows that have drawn me to HBO and Netflix. I don't just randomly watch whatever's on HBO--I have it for Game of Thrones, Silicon Valley, etc.

My use of Netflix is a little more dual-purpose though; there's certainly Netflix-produced shows that I like a lot, but I often find myself nostalgically surfing Netflix to watch old shows like ST:TNG, or to watch favorite movies again. Even though HBO and the different TV providers offer on-demand viewing, there's something about their offerings that makes me much less likely to browse their libraries and watch random content. The paradox is that Netflix's UI is on average worse than theirs, so my preference of browsing shows on Netflix is not a usability thing, it's an emotional thing.

2. The comments about how Netflix has commoditized time since we no longer have to watch shows at a certain time each week. And how this is borne out by their strategy of releasing all episodes at once for shows they produce. I've found, like the author, that releasing them all at once reduces some of the fun and anticipation of watching a new episode and talking about it with your friends. I've also noticed that I tend to binge watch shows when new seasons drop, and tend to fast forward through less exciting parts of the episodes a lot more than normal, since I feel I have to get through the whole season so I can talk about it with my friends. I get really impatient by the time I'm 80% through the season. But at least this shows how aggressively they are committed to giving people their time back and letting them consume the content on their own schedule.

3. I had no idea HBO was producing so many projects at once. Hollywood Reporter said they are working on over 100 projects right now. I didn't consider that Netflix and Amazon could position themselves not just as alternatives to people who couldn't get a seat with HBO, but could also potentially produce the show and get it online faster (e.g. steal good projects from HBO with the promise of fast results, not just taking their rejects). I still think HBO will on average produce higher quality shows for quite some time since they have so much practice at it, but it sounds like they are spread very thin. I would love to read some analysis on how HBO produces their shows and what their org structure is like there.

nocarrier··on Forthcoming OpenSSL releases
This is an excellent explanation of how private key offloading works, thank you.

BTW, you can store private keys in a HSM on your private key server and that's an additional layer of protection, but nobody uses TLS accelerator cards anymore that I'm aware of. You can simply do the relevant crypto on commodity CPUs and still have plenty of throughput.

nocarrier··on Forthcoming OpenSSL releases
Here's some techniques to address #1:

1. Store your private keys in a separate process, either on the same machine or a remote, more secure host. Offload private key operations to the private key service. This obviously requires an encrypted connection between your webserver and the private key service, but you get good gains in security from no longer having the private key in your public facing web server's address space.

2. Another technique you can add is to separate the data plane from the control plane on your public facing webserver--that way you have a stripped-down process that is just handling the low level reading and writing of buffers to the wire, and have a fast pipe connecting it to a separate process doing all the HTTP logic. That way, you can lock down the data plane service and make it harder to exploit since it has a much smaller surface area than a full fledged web server.

nocarrier··on DragonFly BSD 4.2 released
It's cool to hear someone using it, but it's still doomed to be relegated to use inside private networks versus being used as a general use internet transport protocol, which was the point I was making.
nocarrier··on When a Company Is Put Up for Sale, in Many Cases, Your Personal Data Is, Too
I'm guessing everyone who works at a big company and used Secret was hoping that said big company didn't acquire Secret's assets when they folded.

It does always make me think twice before committing to using a startup's product, you never know who is going to fail and sell assets to the highest bidder, or who is going to get acquired by a big company whose ideals you don't like.

The problem is that oftentimes, the data is one of a company's most valuable assets, so you can't just make it off-limits since that would erase a lot of the value. I just wish there was a better legal framework to handle these kinds of situations.

nocarrier··on DragonFly BSD 4.2 released
I'm in favor of Dragonfly removing SCTP. It was okay on paper but never saw adoption and was basically doomed after most firewalls were configured to block it--this is why people are building their own transports on top of UDP these days. It would almost be impossible to launch a new IP magic protocol number now.

I dig the other stuff that Dragonfly is doing too. I've been a longtime BSD fan and I like that there is a strong-willed counterpoint to FreeBSD with a different architectural philosophy. The pressure pushes FreeBSD forward faster than if Dragonfly wasn't around.

nocarrier··on FreeBSD Support for Leap Seconds
Google shared some details about how they do leap smearing and the adjustments were smeared over a long enough time to be small enough to not affect their applications. They've been doing this for a while with no ill effects AFAIK. I'm aware of other people patching ntpd to do similar stuff as well.

If you can get the smear to be gentle enough to not affect your apps, it's a nice workaround for the general problem. There's just so many components running inside a modern infrastructure, and it's hard to get them all to behave correctly during a traditional 61-second-minute leap second transition.

And that 2012 leap second linux kernel bug was pretty epic. AFAIK, people who are smearing didn't experience it.

http://googleblog.blogspot.com/2011/09/time-technology-and-l...

nocarrier··on FreeBSD Support for Leap Seconds
I like that they clearly describe how FreeBSD handles leap seconds, but when I read the headline I was hoping they were announcing support for leap smearing.
nocarrier··on Alex Stamos to Become CSO of Facebook
Yeah, you're right. We certainly had different problems when Joe came on board and he was a great fit to lead us through fixing them. And I'm glad him and Mat and others are working to fix similar problems with Uber.

I just commented because I wanted people to know that it wasn't like there was no software focus before Joe left--things weren't as binary as that. The software focus shift had been in progress for some time before he stepped down.

And yeah, feels good to talk about FB with no filter. :-)

nocarrier··on Alex Stamos to Become CSO of Facebook
Respectfully, you don't sound very well informed and your reply is pretty emotional and negative. Did you actually work on or with FB's Security team? If so, when?
nocarrier··on Alex Stamos to Become CSO of Facebook
It's true that Joe's background is on the legal and policy side of things, but he presided over the softwareification of Facebook's security team. I worked closely with him for many years and he saw the need for more technology and automation; so that they could be a security team focused on not just incident response, but excellence on the technical front too. Joe and I spent a lot of time talking about technology and where things should be going, and he had his own well-formed opinions and goals. And he often deferred to other experts as a good leader does too.

I'm excited to see what Alex does since his credentials are stronger on the technology front, but the team's shift to a technology focus has been happening for a long time. I view this as another hugely positive step in that direction.

nocarrier··on Alex Stamos to Become CSO of Facebook
I'm excited about this. Facebook has aggressively evolved its Security org over the years and they have given security a whole new level of focus in the past few years. I was there for 8.5 years and watched them grow and reinvent themselves a few times. I think FB's Security team is one of its crown jewels and it's great to see Alex joining FB. This will shake things up further and will likely lead to more great things.
nocarrier··on Building a Billion User Load Balancer [video]
We didn't get serious about load balancing and create a team dedicated to it until fall of 2011. At the time, Facebook had 800 million active users per month. We had run out of runway on commercial and open source options, and needed a lot more flexibility with how we allocated and moved traffic around. It took a lot of effort to get to where we are today, but the thing I'll tell you is that these huge systems always start with a prototype running on a dev server. And then you get it to run traffic for your whole team, and then the whole company. Then you try it in a cluster, a datacenter, a region, and all of a sudden you've built a load balancer.

It took Proxygen less than 18 months to go from inception at a hackathon to running all of Facebook's production HTTP traffic. And we could have shaved that by a few months if we had made smarter decisions along the way.

nocarrier··on Building a Billion User Load Balancer [video]
It's a simple, fast DNS server that is straightforward to extend if you need to. The genius of Cartographer is that the backend that calculates where to assign traffic is separate from the DNS server frontend. This let FB use a known-good DNS server that got the common DNS protocol stuff right, and gave them more time to focus on all the magic bits in Cartographer's backend.

DNSSEC was never something we were aiming for, so it's fine that TinyDNS doesn't support it out of the box.

nocarrier··on Building a Billion User Load Balancer [video]
This stuff seems basic at first glance, but it's not easy to get right, and you're discounting a ton of effort that went into getting feedback loops and demand prediction to work properly in Cartographer for example. Maybe it's easy to slap together L4/L7 LB and throw a DNS server in front of it at smaller scales, but it's a notable accomplishment to do this at Facebook's scale. Nothing out there approaches Facebook's request rates.

There have been detailed talks about Proxygen's architecture at various open houses in Seattle and Menlo Park, and they should have HTTP2 support working well soon. I'd expect there's going to be more detailed blog posts on Proxygen at some point. Or you can check out the source code too.

Fragmentation isn't really an issue for either Shiv or Proxygen. I don't think they will share the rate but it is very low.

nocarrier··on An Inside Look at Facebook’s Approach to Automation and Human Work
I built Haystack with two other engineers and then founded the Everstore team which works on Haystack/F4/Cold Storage. I moved off of managing Everstore to focus on Traffic in fall of 2011, so F4 and Cold Storage happened after my time, but I worked on Haystack and the storage management layer above it for five years.

There were a few things that were interesting or novel about Haystack. CDNs were only giving us a 80% hit rate on photos, and even back in 2007 when we first starting working on Haystack, that was an immense number of misses to serve from the origin. That heavily informed the design goals of Haystack, which were to have a very I/O efficient userland filesystem that used direct I/O to squeeze as much iops from the underlying drives as possible. We also wanted zero indirect I/O, so we used an efficient in-memory index so that all seeks on disk were to service production reads versus indirect index blocks or other metadata. Lastly, Haystack is very very simple. We made it as simple as we could and eschewed anything that looked too clever. I actually think its simplicity is one of the things that made it Hard. It would have been much easier to design something more complex.

If you have any questions about the Haystack paper after reading it, I'd be happy to answer what I can. Just send me an email.

Regarding the datacenters, you're kind of simplifying it, but that's okay. There's a lot more orchestration, automation, and integration, and I wish I could give you a tour of one of our OCP datacenters so you can see it firsthand. There's plenty of things I wish I could talk about on the datacenter front that I feel are truly amazing and without precedent, but I can't until they are public. There's an excellent chance that they will become public and likely even part of OCP in the future though. I'm really proud of how much of Facebook's datacenter architecture has been shared with the world for free. It gives me warm fuzzies all the time.

Anyways, thanks for the fun thread, and let me know if you ever want to talk storage sometime.

nocarrier··on An Inside Look at Facebook’s Approach to Automation and Human Work
I’ll talk about one abstract thing and two concrete things that I think show that Facebook has been able to solve Hard problems that required the development of new capabilities rarely or never seen before. I find these three things to be very impressive and notable myself, although I am obviously very biased. I should note that this is not an exhaustive list, it’s more just the three examples I am most familiar with and wanted to take the time to write up.

First, the abstract thing. I don’t know if most engineers outside Facebook appreciate how much activity there is on the site each day. Hundreds of billions of likes and comments. Billions of photos uploaded. Trillions of photos consumed. And growing each day.

And this is for an “online application” in the sense that the data is live and constantly changing. We’re not talking about crawling the web, storing it, doing offline processing, and then building a bunch of indices (which is a different but still legit kind of hard). This is an immense amount of live data producing an even more immense stream of live events; trillions and trillions and trillions that need processing, live and on the fly, every day. It is hard to underestimate how difficult it was to build an application and backend that could drive this kind of social platform. In terms of liveness and scale, there really is nothing out there that can touch Facebook, by orders of magnitude. That is a solved Hard problem in my opinion, if not a meta one. But in my opinion, the largest one.

Here’s two concrete things that I thought are good examples of some Hard problems that Facebook has solved:

1. A global media storage platform that each day is capable of ingesting billions of new photos and videos and delivering trillions as well. This includes Haystack, F4, and Cold Storage, which store hot/warm/cold media objects for Facebook. Each storage layer has specialized, custom software running on distinct storage hardware designed to take advantage of the requirements for each layer. Facebook is the largest photo sharing site in the world by orders of magnitude, and they had to build a very custom photos backend to handle the immense load. I’m not even mentioning their terabit class Edge platform which has a global constellation of POPs that accelerates application traffic and caches popular content close to users. Facebook’s global media storage and delivery platform is truly a unique asset.

2. A datacenter architecture focused on high power efficiency and flexible workloads, comprised of custom built: datacenters, racks, servers, storage nodes, network fabric, rack switches, and aggregation switches. Plus some other things that haven’t been made public yet. This architecture let application developers stop working around various physical performance bottlenecks with compute, storage, cache, and network, and instead just focus on optimizing what was best for the application. At the same time, this new architecture also greatly reduced infrastructure costs.

There are other solved Hard problems I can talk about that are public, and others that I wish I could talk about that aren’t yet. However, I’m not trying to exhaustively defend what Facebook has built, I more just wanted to respond and show a few examples of the very special scale Facebook has and some of the extremely hard problems they’ve had to solve.

nocarrier··on An Inside Look at Facebook’s Approach to Automation and Human Work
It would be cool if you could share when you worked at FB and what general area of the infrastructure you worked on. Your statements don’t match the Facebook infrastructure I know. I recently left after spending 8.5+ years on infra, so I saw it both in the really early days, as well as how it is today.

Facebook actually built many major pieces of its infrastructure itself after finding no suitable solution in open source or commercial vendors. Efforts like this were happening even back in the early days of FB infra. I could list out many publicly known Facebook infrastructure projects that solved Hard Problems. And not just in recent years either. :-)

nocarrier··on The Google Stack [pdf]
Carlos, you are mistaken re: Haystack. I don't think it was ever planned to be replaced, it's always been in use as a hot blob storage engine since it was launched. There are three storage layers for blobs at Facebook: hot (Haystack @ 3.6x replication), warm (F4 @ 2.1x), cold (Cold Storage @ 1.4x). We have published papers describing each layer.

Haystack still handles the initial writes and the burst of heavy read traffic when an object is fresh. It has a higher space overhead of 3.6x because it's optimized for throughput and latency versus cost savings. The Haystack tier's size is now dwarfed by the size of the F4 and Cold Storage tiers, but it still handles almost all of the reads for user blobs on Facebook due to the age/demand exponential curve.

After a haystack volume is locked and has cooled down enough, it moves to F4 to bump its replication factor way down for cheaper, longer term online storage. And then Cold Storage is used for the older stuff that gets barely any reads but that you still want to store online for perpetuity.

That is why the team that works on media storage is called Everstore; they take storing people's photos and videos seriously and view it as a promise to keep your memories available forever. It feels really good to see photos from 5+ years ago and have them still work, and someday there will be 30 year old photos and videos on Everstore as well.

Source: I built Haystack with a couple other people and founded the Everstore team. :-)

← PreviousPage 3 of 3