nocarrier

I loved how the Windows 95 install CD had Weezer's Buddy Holly video on it. I remember finding it and thinking it was so cool they did that.

That article on rapid Rust prototyping matches my experience with using Rust as the backend for a web and iOS app. I used clone() and Vec and String and other shortcuts from that article as much as possible since I was building a backend application versus an operating system. It enabled a lot more velocity and made it fun to add features. And it was still blazing fast.

If anyone is considering using Rust and is nervous about lifetimes and bare metal, check out that article and try its guidance. I learned these things on my own the hard way and would have loved to read this article 18 months ago. It's really quite good.

“During the period 2018-20, Space Norway carried out significant security related upgrades to the fibre connection.”

Sounds like someone was tapping cables with submarines?

Once Colin's patches land on FreeBSD and Firecracker, the total boot time for the kernel is under 20ms. Absolutely incredible times that we live in.

It was actually Rainbow's End by Vernor Vinge that really got Zuck excited about the possibilities of VR—he read it a few years before he bought Oculus and talked about it a lot at the time.

I was at FB from 2006-2015, and Scuba caused a phase change in how Facebook's performance was analyzed and how quickly insights were gained. For many of us in Infrastructure, we spoke of the pre-Scuba, post-Scuba eras. It was such a joy to have aggregated realtime performance metrics for our internet scale services and have the ability to drill down on just about any dimension when doing perf investigations. It gave us so much more confidence in our systems, both in our ability to detect issues, as well as pinpoint what was causing them.

I'd also like to echo what spicyj said about the UI, it is very usable. It was common for managers, PMs, and even people in completely non-technical business roles to use Scuba. It was my favorite internal product at FB and the one I miss the most.

Cartographer ingests BGP topology and POP and datacenter health (and other things), and then pushes what is basicially a lookup table to tinydns. None of the dynamism is inside the DNS server itself, which lets it focus on what it is good at--responding to a crap ton of DNS requests per second.

Cost was a smaller factor than politics; the Indian government wanted the private keys for our certs in order to let FB put a POP there. That was an absolute dealbreaker, so we served India from Singapore and other POPs in nearby countries.

Regarding building a datacenter, that's much higher risk since it's a $100M-1B capital investment. I'm guessing both Facebook and Google see too much volatility to make the risk worth it. It could be an expensive paperweight if the Indian government changed their minds.

I joined FB in 2006 and had never been in an open office environment before. It was quite an adjustment. I didn't like working in the office until I had been there for about six months. The office on University Ave was packed tight and didn't have a lot of creature comforts--it was cheap desks arranged in groups of four, some conference rooms, a break room with snacks, and that's it. And I had come from a cubicle farm at Apple.

There were definitely large benefits to the open floor plan back then since we were working on so many things and each person was wearing so many hats. It made staying in touch with people's progress really easy. We were also working 80-100 hours a week back then, so it was good to constantly stay in touch since there was so little structure and process back then.

I have mixed feelings about FB being held up as one of the models for open floor plans today though. While I think it was definitely useful at the time, I think the open floor plan has become less important as tools have gotten better. We didn't have tools like FB Groups, Phabricator, etc, so you had to be within earshot of your collaborators in order to make sure everyone was in sync.

My sweet spot would be team sized rooms if I was starting a company today. I used to be so jealous of teams that had to grab war rooms and ensconce themselves in there with a bunch of laptops and displays. Even though they were working super hard to meet deadlines, they looked happy as can be.

Indeed, but there is something to be said for sane defaults--look at the thread on HN a few weeks ago about SSH cipher defaults. Yes, TLS and the CA ecosystem has its faults, but my stance is that any chat protocol should be using auth and encryption, especially group chat protocols like IRC. I'm less upset now after seeing that the proposed spec for IRC 3.3 includes something similar to HSTS.

I read IRC v3 and assumed they were making major changes to the wire by bumping the major version, which is why I said what I said. I've been using IRC for 24 years but haven't implemented it before, maybe I'll do that now so I can have a better idea of what I'm talking about.

Hmm, I didn't see the 3.3 spec which is still being developed, it looks like they are doing strict transport security as a base extension. So that would address most of my concerns I think. I would still prefer to see TLS required no matter what.

Fair enough, but I feel you're being a little pedantic since there are base extensions described which are required, and optional extensions which are not required. TLS is one of the optional extensions.

So I'll reframe my question and ask: Why isn't TLS a base extension?

Why isn't encryption baked in by default? It's optional from what I can tell. I don't know how you can design a new protocol and not include encryption.

I think you're being overly harsh. I read the entire article and enjoyed it very much, including the author's commentary.

I didn't interpret their tone as trying to be hard-edged or snarky; I felt it was just them honestly showing the process of someone returning home to a place that was incredibly isolated 25 years ago, and then trying to rectify what it's like for these kids in that same town who now have access to so much more information and avenues of communication.

I liked the different asides where they mentioned what was popular in the town 25 years ago versus what the kids like now, and how the kids interests today compare to bigger city tastes. I grew up in a town that's 5x larger than Napoleon but still pretty small, and I see many parallels with this story that match my experiences.

Not sure if you know this, but you can ask Google to blur a Street View address by filling out a form. I'd guess he did this himself versus any sort of conspiracy thing happening.

To your last paragraph, only relying on forgetting the keys works great, as long as you have absolute 100% confidence in the mechanism used to do that. I read your posts on HN often so I know you know you're quite familiar with defense in depth--I feel that user data is one of those areas where it's ok to do more than one thing to protect the data.

That said, there are a number of systems at FB where deleting a crypto key loses the linked data forever--but they still crunch the hard drives just to be really sure. The drive crunching is an incredibly tiny expenditure compared to the massive CapEx and OpEx required to build, stock, and run the datacenters. It's worth it if only for the peace of mind.

Lucid Emacs -> Xemacs was mentioned in a sibling comment, but I remember people being in awe about EB working on those. There were many interesting employees back then--J. Chenault, J. Gehlen, R. Braumeller, F. Eiden, K. Sears, etc. I've lost touch with most of them but I learned a lot from them.

I started in October 1997 and did Perl work on backend systems, but I had a few friends who worked on the frontend. They used to talk about how Catsubst had a very small set of verbs and that everything was a function, even arithmetic, and that prefix notation was used for all macro calls. They thought it was Lisp-inspired, but they were also pretty junior programmers too. I never worked with Catsubst myself, so my information is second-hand, while yours is zero-hand since you wrote it. :-)

Thanks for all the work you did on Amazon's infra!

There was more Lisp influence than just the CS macros you mention here. Eric Benson was a major contributor to Lucid Emacs and many early Amazon engineers had Lisp experience. SICP was practically issued to every new software engineer--it was that popular.

The template language for Amazon's website at the time (catsubst) was a Lisp-inspired prefix notation language with a relatively small number of functions. For example, there was an add macro but not a subtract macro, so to do A-B, you had to say ${add A -B}. I might have the exact syntax wrong since it's been 20 years.

The two AsiaBSD papers linked from the post are good for more detail. I was a little surprised they had to hack sendfile to do the crypto in the kernel in order to get the throughput they're used to with http, but the reasons are explained in the papers.

However, I'm quite surprised Netflix went with Intel's ISA-L library for AES-GCM given Intel's perf gains were so very marginal compared to BoringSSL. I would have gone with the library that had more eyeballs on it, and in general I'd give Google the edge over writing solid, secure code than I would Intel.

People who watch a football game Sunday night and then talk shit at work on Monday about all the things they would have done differently if they were coach/quarterback/etc.

I'm former FB. To your search question, I hope they build a high quality, accurate search product someday, but they've made a few attempts at search over the years and none of them have really panned out so far. Graph search was retired this year and now it's hard to discover things with search even when you know what you're looking for. And even before, graph search wasn't that accurate, it often didn't include posts in the results that it should have. It's a hard problem, but I think FB could solve it if they want to.

Ah cool, that's similar to what we did. I think http2 gets a bad rap, we saw a lot more positives than negatives from it. You should do a writeup after you test sctp and quic, I'd be pretty interested in reading it and I bet others would too.

The list of firewall-punchable level 4 protocols is frozen in time to just TCP and UDP, and it's sad at first glance, but it's not as bad as we think:

* Most ISPs don't mangle UDP, and if you care enough about perf to be using UDP in the first place, you can afford to figure out who mangles UDP and fall back to TCP for those networks. It's a very small number of networks.

* Any TCP/SCTP/etc inspired protocol you can think of can be implemented on top of UDP. Its header is only eight bytes long and it adds minimal overhead. UDP socket semantics are simple enough that you can treat them as a slightly smarter IP socket and build your protocol as if you were running directly on top of IPv6.

* There's a huge advantage to being able to implement your protocol in userland since you get fine grained control over congestion and connection logic. There's a huge disadvantage too, since it's actually hard to implement that stuff yourself. But for the people who are large enough to build their own protocols (FB, Google, etc), it's worth it.

* You can even encapsulate SCTP inside UDP to punch through middleware firewalls if you want to. I don't know of anyone doing that for real in the wild, but there's a RFC for it (RFC 6951).

This works for your use case which is great, but in my experience, you get gains with full http2 between edge and origin that you miss out on if you demux or fall back to http 1.1. We saw slightly better median latencies and much better >90th percentile latencies with full http2 from edge to origin, versus when we did http2 to edge users and http 1.1 to origin.

At Facebook, we had clients talk http2 to edge nodes near the users, which then edge terminated TLS and opened up the request to figure out which upstream was best to forward the request to. We kept http2 conn pools to each upstream from a given edge location. We also had tunable amounts of domain sharding so that the client could open N http2 conns at once to a given edge hostname (cdn[1-4].facebook.com vs just api.facebook.com or www.facebook.com). This helped with head of line blocking, assuming that your ISP wasn't hopelessly over-congested. In that case, no amount of sharding will help you, and sometimes it even works against you depending on what tcp middleware your mobile ISP is using.

We did different TCP settings for edge<->origin conns than edge<->user which let us balance between latency and throughput for origin traffic as needed, and we did some interesting work tuning edge tcp settings differently for different user networks too. Things like determining the best congestion window for a given ISP's users, how large to set the send buffers, etc.

Admittedly, this work was much easier since we had good http and tcp instrumentation, and we had high quality links between edge and origin. I just wanted people reading your comment to understand the tradeoffs. We saw a measurable difference in perf with a full http2 stack that made up for the extra complexity of running http2 top to bottom.

I'm former FB Infra and agree with your first point re: motives. There's a lot of true believers working in the security space at Facebook, and I have tons of respect for them. I used to work closely with many of them.

However, scaling and/or capacity is not the reason E2E encryption isn't applied on all messages. The crypto operations are relatively trivial in terms of cpu.

This comment summarizes what FB's CSO said about why they are not launching E2E broadly yet. It boils down to usability concerns. Sounds like they are working on it:

https://news.ycombinator.com/item?id=12055567

Feature phone is actually oldspeak. It's what we called cell phones with features back when most cell phones didn't have features. Then smartphones came along and reset the expectations for what features a phone has, leaving feature phones in the dust. But there was a time where feature phone meant something positive.

The Erlang components of Facebook Chat were replaced with C++ several years ago.

The two main reasons I remember are A) it was hard to maintain a group of engineers with acceptable competency in Erlang over time and B) the C++ code offered faster and more consistent performance albeit with somewhat less scalability in terms of sessions per host. We just added X% more servers to the channel pools and were happy to have the chat services in a language where more FB engineers could contribute. There's been a lot more changes in the architecture than just moving to C++ though, so it's hard to do a direct comparison between the products.

This doesn't take anything away from WhatsApp though, who has built a strong product and infrastructure on top of Erlang.

Philip is not unique, FB engineering managers of all levels and tenure move into development roles. It's a relatively common occurrence; I'd guess around 5% of managers switch back to coding each year.

I disagree with this approach.

My policy as a tech lead (and then manager) was to spread the mundane tasks across the team equally. This let everyone experience the pain points and usually led to faster elimination of technical debt, since the senior developers would see patterns in the pain and find ways to address it. (Junior engineers often don't have the experience to recognize bad patterns or bad smells and would happily plow through a bunch of mundane tasks that could be done better with a higher level solution.)

It helped that we had a Tech Debt Week every 8-10 weeks where we took a week to fix problematic code and harden our deployment and automation processes. It was hugely popular with the engineers, we gave out trophies for the best fixes based on team vote.

The "share the love, share the hate" approach also meant that new junior developers were given challenging tasks as soon as they joined the team (and were paired with a senior developer for support). If they excelled, we gave them even harder things to work on. If they struggled, we found easier projects for them to work on and coached them through it so they didn't feel bad about their struggle--we always said it was better to try challenging problems and fail than attempt something safe and easy. Either way, we started new team members with the hard stuff instead of ramping them up slowly on the easy stuff.

The same policy applied to interns. They got challenging and impactful projects, no busy work or work the team didn't want to do. This meant we had a huge return rate on our interns and most of them joined our team when they came to work full-time.