Hacker News Reader

SaveTheRbtzon Migrating Dropbox from Nginx to Envoy

That is indeed true. But, I remember the time when we were rolling out nginx back in 2000's and exactly the same thing was said about Apache.

SaveTheRbtzon Migrating Dropbox from Nginx to Envoy

We did require some of them[1]. Esp. painful were Transfer-Encoding quirks, and some dances around old HTTP/1.0 backends and request buffering.

Compared to NGINX though, it was relatively easy to push these fixes upstream. Community is very welcoming to outside contributions.

[1] https://dropbox.tech/infrastructure/how-we-migrated-dropbox-...

SaveTheRbtzon Migrating Dropbox from Nginx to Envoy

I think the best slice of who's migrating to Envoy can be observed via EnvoyCon talks[1][2]:

* Lyft (of course)

* Spotify

* Stripe

* Square

* eBay

* Yelp

* Pinterest

Plus the support from major cloud providers: Google, Microsoft, and Amazon.

[1] https://envoyconna18.sched.com/ [2] https://envoycon2019.sched.com/

SaveTheRbtzon Courier: Dropbox migration to gRPC

There is a good discussion of LIFO queue behaviours on the corresponding reddit thread[1], featuring the Google's SRE book, Van Jacobson quotes, and real world examples.

[1] https://www.reddit.com/r/programming/comments/adwhk3/courier...

SaveTheRbtzon Courier: Dropbox migration to gRPC

Our train of thought goes like that:

* in non overloaded case LIFO/FIFO make no difference[1].

* in an overloaded case LIFO has better reliability properties, e.g. no standing queue and therefore faster recovery times.

There are alternative approaches there to the problem: Facebook queues switch from FIFO to LIFO when overload is detected[2]. Airbnb uses the same approach[3]. We tend to stick with a simpler way of using LIFO by default.

[1] Given that one does not care about message ordering. If order matters, as mentioned in the post, one can use CoDel or use LIFO only between different connections, maintaining FIFO within each connection.

[2] https://queue.acm.org/detail.cfm?id=2839461

[3] https://medium.com/airbnb-engineering/building-services-at-a...

SaveTheRbtzon Dropbox traffic infrastructure: Edge network

Ouch, that was embarrassing -- thanks for spotting =) Sorry about that -- ESL and stuff. Editors did a hell of a job fixing our English, but errors still slipped in. We'll be fixing grammar (and probably add link to the presentation PDF) later this week.

SaveTheRbtzon Dropbox traffic infrastructure: Edge network

The main principle here is to reuse as less infra as possible: different dns provider, different hosting, different cert authority, etc. So, debug.github.com is probably worse, than github-debug.com (github.debug should be fine though, and even slightly better than github-debug.com)

SaveTheRbtzon Dropbox traffic infrastructure: Edge network

It really depends on the clients, here is an excerpt from the article:

> Here we also need to mention the myriad embedded devices using Dropbox API that range from video cameras to smart fridges which have a tendency of resolving DNS addresses only during power-on.

SaveTheRbtzon Dropbox traffic infrastructure: Edge network

Early edge network was indeed prototyped in AWS: a simple setup with ELB in tcp mode, nginx and proxy_protocol.

Once we've got all data from that experiment: performance, cost, and flexibility included, we've decided to start building our own PoPs.

SaveTheRbtzon Optimizing web servers for high throughput and low latency

> It's not like updating this stuff is without risk.

Same goes for kernel, libc, all other libraries, and pretty much anything else. This is not an argument for not upgrading them though.

Also, sadly, almost all firmware updates fix a "problem [that] points to a specific piece of hardware" that you use.

SaveTheRbtzon Optimizing web servers for high throughput and low latency

I've mentioned it very briefly in a Network Stack part, around the "upgrade your kernel" warning:

> and I’m not even talking about IW10 (which is so 2010)[1]

[1] https://developers.google.com/speed/protocols/Increasing_TCP...

SaveTheRbtzon Optimizing web servers for high throughput and low latency

Physical locations matters a lot for uploads, mostly because client TCP stack is way less sophisticated.

For downloads high RTT can be mitigated by a congestion control that ignores constant packet-loss rates (which are common for high-rtt paths). Other tricks that you can try: fq+pacing and newer kernels with more sophisticated recovery heuristics.

SaveTheRbtzon Optimizing web servers for high throughput and low latency

I've tried to:

1) give an bcc/perf example to check the need for tuning and verify its effect.

2) give code/docs/paper reference as an embedded link.

3) give a generic monitoring guideline at the start of a "chapter".

Seems like I've (at least partially) failed. I'll do better next time.

SaveTheRbtzon Optimizing web servers for high throughput and low latency

When I was at Yandex, we were a FreeBSD-CURRENT shop (FreeBSD 9 at that time). So, that said, FreeBSD will had exactly the same[0] issues on the low/mid levels, just not all of them had a good solution[1][2].

[0] https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerfo...

[1] https://wiki.freebsd.org/NetworkPerformanceTuning

[2] https://wiki.freebsd.org/TransportProtocols

SaveTheRbtzon Deploying Brotli for static content

By "dict" I meant the data we are compressing: these are basically dictionaries for "English to X" translation.

What was saying is that there is a strong correlation between size of the data I was compressing and compression ratio improvements over gzip.

SaveTheRbtzon Espresso – Google’s peering edge architecture

This is a good talk about decomposability of control plane and creating proper abstractions for it.

PS. "The ability to master complexity is not the same as the ability to extract simplicity" is a good takeaway. PPS. This is a part of EE 122: https://inst.eecs.berkeley.edu/~ee122/fa12/class.html PPPS. PDF for SDN lecture: https://inst.eecs.berkeley.edu/~ee122/fa12/notes/25-SDN.pdf

SaveTheRbtzon Deploying Brotli for static content

This is indeed what was used there. Here are all the matplotlib docs rendered with XKCD: https://matplotlib.org/xkcd/gallery.html

SaveTheRbtzon Deploying Brotli for static content

The first draft of the article actually had that reason, but there is also a strong correlation between the size of the dict (these dicts are almost 1Mb, while other languages are closer to 500kb) and compression ratio improvements. Therefore I've played it safe and attributed it to the window size.

Though for languages like Korean and Chinese (whose size is more inline with latin languages) we see 27.5% improvement, which is most likely due to context modeling.

Therefore I assume ratio improvement is split ~50/50 between these two. It was easy to verify that by compressing data with `brotli --window 15` and comparing ratios there, but I was lazy there. I'm sorry.

PS. I've also skipped NFC/NFD part of the post which is very interesting for Korean, where NFC normalized text occupies 30% less space. It also gives additional ratio 5% for brotli and 15% for gzip.

SaveTheRbtzon Making Chrome on Windows Faster with PGO

The article is a bit shallow. It would be nice to see:

1. What flavours of PGO optimizations were applied? What was the isolated impact of each one of them on both speed and size of the code?

2. What tests did they use to "guide" PGO?

3. How did they analyze PGO results(except for these three tests that were provided)? I assume they did not blindly trust it, therefore there should be a way of visualizing differences between two binaries with millions of lines of code.

4. How did PGO affect crash statistics?

SaveTheRbtzon CVE-2015-0235 – GHOST: glibc gethostbyname buffer overflow

nginx on most supporting platforms (`NGX_HAVE_GETADDRINFO && NGX_HAVE_INET6`) uses `getaddrinfo(3)`.

SaveTheRbtzon Lightspeed – A Browser Experiment

Looks a lot like Yandex.Browser[1]. Also built around Search, Favorites, Quick Links, Synchronization and Security.

[1] http://browser.yandex.com/desktop/

SaveTheRbtzon New algorithm shakes up cryptography

DJB says following about all non-prime fields [1]:

  > Is ECDLP broken for non-prime fields?

  No. However, the security story for non-prime fields 
  (e.g., binary extension fields) is more complicated and 
  less stable than the security story for prime fields, as 
  illustrated by 1998 Frey, 2002 Gaudry–Hess–Smart, 2009 
  Gaudry, and 2012 Petit–Quisquater.

  2006 Bernstein stated that prime fields "have the virtue 
  of minimizing the number of security concerns for 
  elliptic-curve cryptography". Similarly, the Brainpool 
  standard and NSA's Suite B standards require prime fields.
  There is general agreement that prime fields are the safe,
  conservative choice for ECC.

[1] http://safecurves.cr.yp.to/field.html

SaveTheRbtzon New algorithm shakes up cryptography

And what does it mean in Standard [FIPS186-2] [SEC-2v2] terms? That one should avoid using sect{163,233,239,283,409,571}k1 and sect{163,233,239,283,409,571}r1 curves? But there are also rumors that secp{192,224,256,384,521}r1 are backdoored by NSA. This leaves only secp{192,224,256,384,521}k1 which are considered "unsafe" by Daniel J. Bernstein [safecurves].

So the only hope is to wait for [Curve25519] to be widely supported?

PS. Bitcoin using secp256k1 which is Koblitz GFp curve.

[FIPS186-2] http://csrc.nist.gov/publications/fips/archive/fips186-2/fip...

[SEC-2v2] http://www.secg.org/download/aid-784/sec2-v2.pdf

[safecurves] http://safecurves.cr.yp.to/index.html

[Curve25519] http://cr.yp.to/ecdh.html

SaveTheRbtzon HTTP 2.0: GTFO (General Termination of Future Operations)

I still can't decide whatever it's childish or fking awesome!

SaveTheRbtzon Chrome Is The New C Runtime

With the same reasoning you can use nginx as a C runtime.

SaveTheRbtzon The babies who nap in sub-zero temperatures

As for born in Siberia my first thought was "Isn't it normal for every child to have a nap in -35C or so?" It was rather normal for our schools back there to be closed once or twice per winter because of extreme cold, but kids where just playing hockey on the school grounds instead =)