These stories of kids breaking things at work reminded me of ultimate kids-at-work-gone-wrong story when a Russian airline pilot let his kids fly the plane. Such a tragic story.
682 karma · joined June 7, 2012
Amazon 1997-2004 (supply chain, product catalog, website engine, service frameworks, image processing and serving)
Apple 2005-2006 (product catalog, image rendering)
Facebook 2006-2015, 2020-2021 (storage, networking, incident management, site performance & investigation, internal & external threats)
I have a question after reading the Vault overview here:
https://www.backblaze.com/blog/vault-cloud-storage-architect...
I'm curious what the bandwidth demand is when an entire host fails and you have to drop a new replacement host in, since you would have 60 Tomes in the Vault that need to be rebuilt at once. You have at least two other parity copies when a host dies (assuming no other drives in the affected host's Tomes are dead on other hosts in the Vault set), so I'm guessing you can afford to wait the handful of days it would take to rebuild the host. I'm still curious to know what rebuild speed you plan for, since I'm guessing you'd be looking at 40Gbps NICs eventually. I was surprised to see only 2x10Gbps.
However, then it goes on to say a few things that I think are harder to defend. The author talks about how it's hard for Apple to build good services since they're so focused on achieving perfection with a tightly integrated device that has a new release once or twice a year. Basically, the thesis is that since the market penalty for releasing a bad device is so high, Apple has thrown a lot of resources into getting the device right the first time, which makes it very hard for them to think or operate at the tempo you need when operating services:
"You only get one shot to get a device right, so all of Apple’s internal rhythms and processes are organized around delivering as perfect a product as possible at a specific moment in time."
This isn't right in a couple ways. First, Apple has released all kinds of sub-par hardware in the past. And will likely do so in the future. Apple says they are looking to achieve perfection, and it sounds noble, but it's more of an iterative process than this article makes it sound like.
Second, many of a device's features, especially the integration that the author lauds, are actually in software. Particularly the integration pieces that make the device seem like magic. And those are updated often as one can see with the release tempo of iOS updates. The pattern has been the same for a long time--release a bunch of new features in iOS X.0, then do point releases to fix bugs, plug security holes, and make minor tweaks to fix annoying things and make the user experience feel more smooth.
The other conclusion I don't agree with is that the author says that in order for them to build better services, they need more accountability, which would be achieved by tracking profit and loss for each service and making the leaders financially accountable. I agree that accountability is something that's required to create strong services, but it's not the primary lever I would use. And I certainly wouldn't bring P&L into the accountability equation since that incentivizes the different service orgs to grow adversarial relationships. It's old company thinking IMO.
Instead, I feel you need to build better services by building a better infrastructure culture. A lot of people look at situations like this and look for punitive or regressive measures to fix the problem, i.e. using the stick instead of the carrot. I think however they would build better services by focusing on other levers like open communication (the heavy siloing meant a lot of duplicated work), being open to criticism (lots of politics and defensiveness meant fiefdoms and grudges were created), a focus on instrumenting and measuring performance of services to have the right picture for how everything is working together (instrumentation and visualization of performance was terrible), a culture of learning from mistakes instead of assigning blame, having embedded SRE-like engineers who focus on production quality and availability instead of having separate second-class ops teams, etc. I could go on, but I have a really strong objection to boiling all their problems down to not making each service unit financially accountable. That won't fix things, it will instead make the internal culture on the infrastructure side much worse.
However, then it goes on to say a few things that I think are harder to defend. The author talks about how it's hard for Apple to build good services since they're so focused on achieving perfection with a tightly integrated device that has a new release once or twice a year. Basically, the thesis is that since the market penalty for releasing a bad device is so high, Apple has thrown a lot of resources into getting the device right the first time, which makes it very hard for them to think or operate at the tempo you need when operating services:
"You only get one shot to get a device right, so all of Apple’s internal rhythms and processes are organized around delivering as perfect a product as possible at a specific moment in time."
This isn't right in a couple ways. First, Apple has released all kinds of sub-par hardware in the past. And will likely do so in the future. Apple says they are looking to achieve perfection, and it sounds noble, but it's more of an iterative process than this article makes it sound like.
Second, many of a device's features, especially the integration that the author lauds, are actually in software. Particularly the integration pieces that make the device seem like magic. And those are updated often as one can see with the release tempo of iOS updates. The pattern has been the same for a long time--release a bunch of new features in iOS X.0, then do point releases to fix bugs, plug security holes, and make minor tweaks to fix annoying things and make the user experience feel more smooth.
The other conclusion I don't agree with is that the author says that in order for them to build better services, they need more accountability, which would be achieved by tracking profit and loss for each service and making the leaders financially accountable. I agree that accountability is something that's required to create strong services, but it's not the primary lever I would use. And I certainly wouldn't bring P&L into the accountability equation since that incentivizes the different service orgs to grow adversarial relationships. It's old company thinking IMO.
Instead, I feel you need to build better services by building a better infrastructure culture. A lot of people look at situations like this and look for punitive or regressive measures to fix the problem, i.e. using the stick instead of the carrot. I think however they would build better services by focusing on other levers like open communication (the heavy siloing meant a lot of duplicated work), being open to criticism (lots of politics and defensiveness meant fiefdoms and grudges were created), a focus on instrumenting and measuring performance of services to have the right picture for how everything is working together (instrumentation and visualization of performance was terrible), a culture of learning from mistakes instead of assigning blame, having embedded SRE-like engineers who focus on production quality and availability instead of having separate second-class ops teams, etc. I could go on, but I have a really strong objection to boiling all their problems down to not making each service unit financially accountable. That won't fix things, it will instead make the internal culture on the infrastructure side much worse.
Traditionally, it is also known to have a more scalable network stack than Linux, as well as super solid virtual memory and i/o systems.
I still like FreeBSD a lot and donate to the FreeBSD Foundation so they can keep doing what they do, since I think competition is great. But I'll say that in my opinion, the performance and stability differential between FreeBSD and Linux is mostly gone practically speaking. It's still much better organized than Linux distros, and the man pages are great, but I feel that unless you are looking for a BSD licensed OS, Linux is now "good enough" for heavy duty networking and i/o, where in the past you might have chosen FreeBSD for that role. There are some notable exceptions (Netflix uses FreeBSD for their CDN for example), but I do think that Linux has caught up in most areas where FreeBSD previously had a strong lead.
I still have a FreeBSD machine though, cause I love the OS to death. Happy user since 2.1.7.
I don't remember how long we ran it for, but IIRC we had enough candidates to stop after a week or two. I'm guessing something like 100-200M cpu hours?
I don't know if the story behind the facebookcorewwwi.onion domain name itself has been talked about much, but we wanted a memorable name for the domain so we took a new cluster that hadn't been put in production yet and threw something like 500k cores at brute forcing onion names till we had a memorable domain name. Alec had a script that looked for hashes that started with facebook and then he picked the one that seemed to fit the most. And that's how we have facebookcorewwwi.onion now.
Here's more info on payments in Messenger, but note that this doesn't include info on the platform stuff and how payments interact with it since it's brand new:
https://www.facebook.com/help/messenger-app/750020781733477/
http://www.theverge.com/2016/4/8/11392138/spacex-landing-suc...
To put it another way, traction factors a lot more heavily into the best route around a corner and therefore demands fewer instances of full-on/full-off throttle and braking that you describe.
On the hacker front, I noticed a lot of patterns--they had a lot of drive, were self-taught (obviously), and had other things about them that were remarkable. One can play any song on the piano that he's heard just once before. Another is a multiple-time CTF champion. Another is the best performance engineer I've ever met, and who can diagnose and fix site outages 10x faster than anyone else.
There's a difference between hackers and engineers though. I found that the engineers were good at breaking projects into milestones and driving their teams toward them, e.g. steady state development when conditions are good. And I found that the hackers excelled at exceptional or emergency situations like debugging complicated multithreaded code on the fly or figuring out which switch in a datacenter was corrupting packets for production traffic. There were exceptions in both directions (hackers who were great technical leads and engineers who were great detectives), but this was a pretty general rule I saw over the years.
I also worked with a number of people without degrees that weren't very good, so it's not like the lack of a degree makes you a hacker or makes you good at your job. You have to be very driven when you're self-taught and sometimes have to work harder than your peers who have a more evenly balanced education.
I ended up being a bit of a wildcard myself--I'm a self-taught hacker and did lots of cool debugging and detective work for 14 years, and then I was a manager for 5 years. I think hackers can make interesting managers since they are willing to upend traditions and try new things with their teams. I did a lot of that.
https://www.bostonglobe.com/business/2016/03/23/documents-re...
Edit: Here's the worst stuff the article mentioned, I figured it would be good to highlight the relevant quote. These are felonies.
The documents also say there was an effort “to obtain sensitive information on individuals with access to the book’s transcript, or control of the publishing deal. The information found was then used as leverage in an attempt to prevent the book from reaching the market place.”
The report also mentions “tactics such as email hacking and extortion” in the attempt “to railroad the book.”
Of course, we have to go off of what Oculus and FB says, but they are working to create an ecosystem and market for VR experiences and seem to be pushing VR forward wherever they can; look at Oculus Studio and the work they're doing to create interactive movies, as well as the software deals with partners like Samsung's Gear VR. Plus, VR is one of the three stated future pillars of FB (the other two are AI and Internet-for-all).
I'd be willing to put money on Oculus being around and still being a viable business for FB in three years, even if the Oculus Rift hardware is not a commercial success. If the hardware is not popular, they will still be pushing VR forward in other ways. VR is a huge opportunity and is likely to be the entertainment medium of the future. I sincerely doubt FB is going to be overreacting and shutting down Oculus the way you describe.
"When talking about a team that is not producing work fast enough, look at the records. Look at the team chats and emails, look at the tickets, look at the repository code reviews and checkins."
Early BBS culture was really something else--we'd do meetups at the local Godfather's Pizza on Saturday afternoon and knew each other by our handles. We'd trade notes on new BBSes that had cool downloads or interesting message boards, and give each other advice on which companies had the best long distance plans. I remember doing math to calculate how many 3.5" disks worth of data I could download from a given BBS in another area code--IIRC it was something like $5 per disk with a 9600bps modem. I didn't even try downloading full games until I had a fast enough modem because downloading a floppy at 2400bps meant you'd blow out your daily time allocation before the download finished. What an era.
Thanks Chuck.
You can setup IPsec tunnels on your application nodes and use that to get transparent encryption of all traffic, but that is pretty complicated to configure and manage. It's much easier to just have your different applications use an encrypted and authenticated transport when talking to each other over the network, whether they are in the same datacenter, or talking to a remote datacenter. This is how Google and Facebook do their cross datacenter encryption.
This is almost four years old, shouldn't it have (2011) in the title? I don't think Facebook even does this anymore.
Yes, it's still a problem, because then you have to depend on the whitelist staying valid and never having an admin accidentally turn it off. And you also have to depend on none of the machines on the trusted IPs being compromised either. And you have to depend on many other things not happening as well.
Instead, you want what is called defense in depth--several layers of security so that an attacker needs to breach several defensive layers in order to get access to what they're looking for. Relying on just an IP whitelist as a single layer of defense is not considered to be a good practice.