bdd

I truly appreciate Backblaze publishing these but it's paramount to remember their disk usage pattern is fundamentally different than yours. This data is almost irrelevant to majority of their audience trying to deduce something out of it to help their buying decisions.

In addition to usage patterns, there's a lot more that contributes to mean time between failure of hard drives; like operating temperature (ambient, fluctuations, max), relative humidity, vibration and alike.

There's also well researched and published literature around this.

From Microsoft 2016: Finds relative humidity to be the dominant factor for disk failure https://www.usenix.org/system/files/conference/fast16/fast16...

From UToronto 2012: Finds no correlation with DRAM failures, nor high temp and disk failure but concludes temperature variability affects disk reliability more than maximum operating temperature http://www.cs.toronto.edu/~bianca/papers/temperature_cam.pdf

From Google 2007: cannot find strong correlation between failure rate and temperature or utilization. https://static.googleusercontent.com/media/research.google.c...

From Microsoft 2013: focuses on operating temperature being the driving factor of failure, more than load. http://www.cs.virginia.edu/~gurumurthi/papers/acmtos13.pdf

If you found this interesting but didn't fully get some of the things explained, I can recommend checking out W2AEW's video where he explains things on paper and illustrates them on the bench in under 12 minutes. https://www.youtube.com/watch?v=81C4IfONt3o

Alan Wolke (W2AEW) is a prominent YouTuber making videos on electronics, ham radio, and related topics. He is a long tenured Tektronix employee and an outstanding technical educator.

Try this: Incumbent products are so bad and so expensive to the degree it’s noticeable for a company this rich. Noticeable enough that it decides to spend money in engineering and highly custom hardware manufacturing is still a better bang for buck.

I see some replies from GH employees. Do you intend to support /v1/search or /v2/_catalog APIs? Would be nice to have access to at least public image listing like Docker Hub's.

You can further Emacsify or customize text controls to your heart with a `DefaultKeyBinding.dict` file.

This what I've been using for ~8 years. Apple removed a few functions here and there but still plenty are available. Reference URL in the file comments.

    {
        /* Additional Emacs bindings:
         *
         * "<key combination>" = "functionName:";
         *
         * Key Modifiers:
         *   ^: ctrl
         *   ~: option
         *   $: shift
         *   @: command
         *
         * Instance Methods of "NSStandardKeyBindingResponding":
         * https://developer.apple.com/documentation/appkit/nsstandardkeybindingresponding
         *
         * Install under ~/Library/KeyBindings/DefaultKeyBinding.dict
         *
         * Download directly:
         * curl --create-dirs -o ~/Library/KeyBindings/DefaultKeyBinding.dict THIS_URL
         */


        "^l"  = "centerSelectionInVisibleArea:";
        "^/"  = "undo:";
        "^ "  = "setMark:";

        "~f" = "moveWordForward:";
        "~b" = "moveWordBackward:";
        "~<" = "moveToBeginningOfDocument:";
        "~>" = "moveToEndOfDocument:";
        "~v" = "pageUp:";
        "~d" = "deleteWordForward:";
        "^w" = "deleteWordBackward:";
        "~/" = "complete:";
        "~t" = "transposeWords:";

        "^x" = {
            "h"  = "selectAll:";
            "k"  = "performClose:";
            "^x" = "swapWithMark:";
            "^m" = "selectToMark:";
        };
    }

I run an apu2c4 at home, with 1 Gbps internet connectivity. It’s a two legged setup, one leg to the ONT, and one leg to the switch carrying dot1q tagged VLAN traffic. It never fails to saturate that internal interface and does that with ~20-30% cpu idle time. I also run few small processes like DNS, NTP, syslog receiver in their own containers, serving the home. The only time this machine struggles is when my wife is out in a library, coffee shop with good connectivity and her VPN traffic (wireguard) to home is above 200 Mbps and at the same time I’m downloading from somewhere that can feed me at least 800Mbps. In this case it cannot saturate the gigabit interface because CPU is too busy.

Linux 4.19 kernel. IPtables for packet filtering and IPv4 NAT. ~30% of my traffic is v6 without NAT overhead.

Maybe it wasn't very clear in my comment but "correctness issues" more-or-less literally mean the instruction's result is incorrect. Like you ask 2*2 and it gives you 0xfff1309f. So depending on how you rely on that result, you may crash. Extending on my examples, bad RSA multiplication means failed authentication. Bad AES construction means encrypting things to garbage, impossible to decrypted by the recipient. So, let's say if you rely on a result from such instruction to do memory references, you're definitely going to crash.

“Unable to perform AVX2 instructions correctly under heavy load” is also a common “WTF Intel!?”–inducing phenomenon. I’m certain SREs who work at companies with more than 1 million servers have a bunch of hair pulling stories.

Most (all?) Intel server CPUs in fact decrease clock speed when executing AVX2 (and some other) instructions to keep things a bit more sane. Vlad from Cloudflare wrote about this, more specific to AVX-512 back in 2017: https://blog.cloudflare.com/on-the-dangers-of-intels-frequen...

Then there is PROCHOT signal. Which is supposed to protect the CPU from getting too hot but keeps getting raised in lopsided AVX2 loads not because CPU is too hot but voltage regulation gets whacked.

You may wonder: what is an example of AVX2 heavy load. RSA multiplication is a good candidate. AES constructions or modes (CBC with SHA, GCM) are implemented in AVX2-BMI2 as well.

I miss the kind strangers who genuinely cared about helping others without being abrasive jerks. I learned a lot on IRC; made many new friends. This was back in 1994–98 when I was growing up in Turkey. The scene was rather small yet still full of kind enthusiasts. Sure there was some passingly “oh you’re a n00b” messing but it wasn’t toxic. EFNet and DalNet were a bit more different. Maybe less kind but still full of helpful people. If you weren’t grossly stepping out of boundaries of netiquette, you’d get treated with reciprocal dignity.

Here's the equivalent in C++, compiled with clang 9.0.0 at optimization level 1. https://godbolt.org/z/nKSG9_

Byte for byte identical assembly.

Anyone can build a frontend. "Java bytecode" is in that list too. It doesn't mean defacto compilers of these languages rely on LLVM.

These are not about abstractions but compile time optimizations. These are also not implemented in the Rust compiler but LLVM. So any any language frontend in front of LLVM would yield the same optimizations. According to Wikipedia the list is:

> [...] variety of front ends: languages with compilers that use LLVM include ActionScript, Ada, C#, Common Lisp, Crystal, CUDA, D, Delphi, Dylan, Fortran, Graphical G Programming Language,Halide, Haskell, Java bytecode, Julia, Kotlin, Lua, Objective-C, OpenGL Shading Language, Ruby, Rust, Scala, Swift, Xojo, and Zig.

Sometimes I think mention of Rust in the title just gets upvotes without even reading the article, here.

It’s from Facebook’s Red Team, which attacks Facebook and tries to move laterally like a real, motivated, and persistent attacker. They build some nice tooling, sometimes more advanced and well thought than what attackers actually use to stay under the radar and accomplish their objectives. This one is one of the many.

You probably want year round low humidity and relatively low temperatures following the seasonal curves.

Oregon's High Desert (https://en.wikipedia.org/wiki/High_Desert_(Oregon)) is a good one.

Apple and Facebook have data centers in Prineville, OR. Google has a GCP data center in The Dalles. Amazon has at least one in Port of Morrow and one in Port of Umatilla.

In fact they can be all lined up on a 236 mile stretch. https://goo.gl/maps/inGGCCk2qBb8Bg5z5

Back in 2011 Facebook published about the efficiency of Prineville data center, mentioning 100% outside air direct evaporative cooling https://engineering.fb.com/core-data/designing-a-very-effici... and efficiency of water use https://www.opencompute.org/blog/water-efficiency-at-faceboo...

...Now in 2019 we have data centers in various locations across the world but efficiency goals are no different if not more ambitious. So something more flexible was developed and now in production. https://engineering.fb.com/data-center-engineering/statepoin... IIRC it doesn't mean we high deserts aren't special anymore. They still are.

Personal experience spanning across my time at Twitter and Facebook, DPDK is definitely faster but it is (/was) a pain in the butt to program and share the NIC with the host. XDP allows you to reuse very many packet parsing / handling capabilities in the kernel while in DPDK world you’re shipping a tiny stack with your app.

Re: Virtual Switch implemented in BPF: see Cilium’s work to connect containers with their BPF based connectors.

The number increases because there are various monitoring tools. Stuff like something that extracts more data from TCP retransmits so we have a better understanding of congestions and errors in the network path. ...or things that simply collect certain system events for security event detection. Imagine, for every certain event, what is injected is counted as a separate program.

On top of these which is common for every machine, service owners can deploy their own BPF programs for specific use cases. In fact our self service tracing tooling is also a BPF program. We talked about it back in 2014 when it did not use BPF https://tracingsummit.org/w/images/6/6f/TracingSummit2014-Tr...

Excuse me while I shill for my employer but we're indeed big fans of BPF at Facebook.

Our L4 load balancer is implemented entirely in BPF byte code emitting C++ and relies on XDP for "blazing fast" (comms approved totally scientific replacement for gbps and pps figures...) packet forwarding. It's open source and was discussed here at HN before https://news.ycombinator.com/item?id=17199921.

We discussed how we use eBPF for traffic shaping in our internal networks at Linux Plumber's Conference http://vger.kernel.org/lpc-bpf2018.html#session-9

We presented how we enforce network traffic encryption, catch and terminate cleartext communication, again, you guessed, with BPF at Networking@Scale https://atscaleconference.com/events/networking-scale-3/ (video coming soon, I think.)

Firewalls with BPF? Sure we have 'em. http://vger.kernel.org/lpc_net2018_talks/ebpf-firewall-LPC.p...

In addition to all these nice applications we heavily rely on fleet wide tooling constructed with eBPF to monitor:

  - performance (why is it slow? why does it allocate this much?)
  - correctness (collect evidence it's doing its job like counters and logs. this should never happen, catch if it does!)

...in our systems.

Not only WhatsApp but there are no files checked into any Facebook source repository with .ex or .exs suffix and looks like Elixir.

This isn't true at all.

Major functions of WhatsApp are implemented in Erlang and they're actively worked on.

Source: Self. Engineer at FB.

look closer to the first image in the linked URL. it's printed on paper and photo is taken with a phone. you can see the flare from an overhead room light or maybe camera flash in the middle and printed paper's wave on the upper left corner.

AmigaOS Workbench 1.2 had an easter egg displaying the title of this post in the menubar. Turns out engineers from Los Gatos office were pissed and included this in the release.

I've long forgotten the key combination and actions to trigger but luckily Amiga History Guide is still up http://www.amigahistory.plus.com/messages.html

Here's the demo of the easter egg on YouTube https://www.youtube.com/watch?v=omMOuyTLmyg&feature=youtu.be...

I don't see the value in lamenting the old days of a few machines where you could actually name them as Middle Earth characters, install individually, log in to one single machine to debug a site issue. The problems were smaller and individual server capacity in respect to demand was in meaningful fractions. Now the demand is so high and set of functions these big companies need to offer are so large, it's unrealistic to expect solutions that doesn't require distributed computing. It comes with "necessary evils", like but not limited to configuration management--i.e. ability to push configuration, near real time, without redeploying and restarting--, and service discovery--i.e. turning logical service names to a set of actual network and transport layer addresses, optionally with RPC protocol specifics. I refer to them as necessary evils because the logical system image of these are in fact single points of failures. Isn't it paradoxical? Not really. We then work on making these systems more resilient to the very nature of distributed systems, machine errors. Then again, we're intentionally building very powerful tools that can also enable us to take everything down with very little effort because they're all mighty powerful. Like the SPoF line above, isn't it paradoxical? Not really :) We then work on making these more resilient to human errors. We work on better developer/operator experience. Think about automated canarying of configuration, availability aware service discovery systems, simulating impact before committing these real time changes, etc. It's a lot of work and absolutely not a "solved problem" in a way single solution will work for any scale operation. We may be great at building sharp tools but we still suck at ergonomics. When I was at Twitter, a common knee-jerk comment at HN was "WTF? Why do they need 3000 engineers. I wrote a Twitter clone over the weekend". A sizable chunk of that many people work on tooling. It's hard.

You're pondering if hiring practices and turnover might be related? The answer is an absolute yes. On the other hand, these are the realities of life in large tech companies. Hiring practices change over years because there's a limited supply of of candidates experienced in such large reliability operations and industry doesn't mint many of them either. We hire people from all backgrounds and work hard on turning them to SREs or PEs. It's great for the much needed diversity (race, gender, background, everything) and I'm certain the results will be terrific but we need many more years of progress to declare success and pose in front of a mission accomplished banner on an aircraft carrier ;)

You are also wisely questioning if turnover might be contributing to these outages and prolonged recovery times. Without a single doubt, again the answer is yes but it's not the root cause. Similar to how hiring changes as company grows, tactics for handling turnover has to change too. It's not like people leave the company, but within the same company they move on and work on something else. The onus is on everyone, not just managers, directors, VPs to make sure we're building things where ownership transfer us 1) possible 2) relatively easy. This in mind, veterans in these companies approach code reviews differently. If you have tooling to remove the duty of nitpicking about frigging coding style, and applying lints, then humans can indeed give actually important feedback on complexity of operations, self describing nature of code, or even committing things along with changes to operations manual living in the same repo.

I think you're spot on with your questions but what I'm trying to say with this many words and examples is, nothing alone is the sole perpetrator of outages. A lot of issues come together and brew over time. Good news, we're getting better.

Why did I move around? Change is what makes life bearable. Joining Twitter was among the best decisions in my career. Learned a lot, made lifelong friends. They started leaving because they were yearning a change Twitter couldn't offer. I wasn't any different. Facebook was a new challenge, I met people I'd love to work with and decided give it a try. I truly enjoy life there even though I'm working on higher stress stuff. Facebook is a great place to work but I'm sure I can't convince even %1 of HN user base, so please save your keyboards' remaining butterfly switch lifetime, don't reply to tell me how much my employer sucks :) I really hope you do enjoy your startup jobs (I guess?) as much as I do my big company one.

If you are bored one day and around Menlo Park, come have a coffee or ice cream at FB campus. You can troll me in person.

geez, tough crowd. do you wanna ten dollar hug?

> How does the fact you worked at those companies relate to #4?

For Facebook I worked on the incident, previous Wednesday. 9.5 hours of pain...

And for my past employers, I still have friends there texting the root causes with facepalm emojis.

#4

I work at Facebook. I worked at Twitter. I worked at CloudFlare. The answer is nothing other than #4.

#1 has the right premise but the wrong conclusion. Software complexity will continue escalating until it drops by either commoditization or redefining problems. Companies at the scale of FAANG(+T) continually accumulate tech debt in pockets and they eventually become the biggest threats to availability. Not the new shiny things. The sinusoidal pattern of exposure will continue.

end-to-ends (notice plural) encrypted images are also served from the CDN.

Was Akamai also down or you assumed Facebook still uses Akamai? If the latter, I can say for the last 2 years Facebook didn’t use Akamai to service any geographic location.

A small nit about Curve25519 vs X25519. The name of the curve is Curve25519, while the name of Diffie Hellman function utilizing the curve is X25519. Same with Curve448 and X448.

That's why the line `TLS curves: X25519, prime256v1, secp384r1` should be `TLS curves: Curve25519, [...] `

I'd impose my pedantry on the Mozilla wiki but it's not accepting account registrations, I guess understandably.