Performance and Usage at Instagram | Hacker News Reader

Performance and Usage at Instagram

engineering.instagram.com

87 pointscgredona29 comments

> We saw app-wide impressions across all mobile platforms increase by 0.7%, and likes increase by 0.4%. This was driven by increases of impressions on all surfaces – for instance, “Explore Photos” impressions increased over 3%, and user profile scrolls increased 2.7%.

I'm curious about this: was it A/B tested? Those figures are so small that they could just be seasonal or organic growth, the side effect of other things (events, weather, etc) that lead to increased engagement in general, and so on.

The technical improvements look great, but attributing them directly to increased user engagement seems a little too fuzzy.

Yep, these were all A/B tested and the results were statistically significant.

Always glad to hear that a big platform uses Django as backend. I wonder if the spec are still the same as the ones listed here? http://instagram-engineering.tumblr.com/post/13649370142/wha...

We have a follow up to that post coming soon, stay tuned.

OOM errors dropping by 30% when you go from 20 comments to 5 doesn't sound like "a job well done", that's more your initial hint to figure the hell out why 20 comments cause memory issues.

I agree. and I'm surprised they made a blog post about this. The details of this seems more embarrassing than it is enlightening.

>Comment bundles are particularly heavy: They contain the comment text itself as well as its id, timestamp, and author metadata (including profile picture). ... Generating profile picture URLs is a CPU-inefficient operation because we must dynamically compute the correct CDN URL. The more comments we load, the more profile picture URLs we need to generate.

Is it loading the entire profile picture with each comment? Not just a small thumbnail?

I think it's only loading the URL to the profile photo image, which is stored in the CDN. Calculating the URL is, apparently, the slow factor. Maybe it requires some type of cryptographic operation that results in heavy CPU use?

Many 'big operations' will use some sort of image-resizer CDN, whether something custom made in-house, off-the-shelf/open source (like thumbor), or a CDN like Imgix.

To prevent anyone from fiddling with the URL to generate thousands of differently sized images, they'll often have a way to 'sign' the URL to ensure people can't do this sort of attack. This can be expensive, depending on how its done I guess.

It could be a hash operation, yes. But would it really be necessary to make it a cryptographic hash? It would explain the performance issues, yes. But who uses cryptographically-secure hashing for use-cases where the answer is "MD5"?

>I think it's only loading the URL to the profile photo image

Sorry, by "it" I should have written "the client." I meant to suggest my question of whether that is where their client OOM errors are coming from.

We load a small thumbnail; it's the extra memory allocated to strings that was the main factor that helped when we reduced the # of comments sent down.

Just this morning I was thinking about making a post here about how I recently had occasion to install Tinder. One of the most interesting things about Tinder (on Android) is that it crashes and won't load images ALL THE TIME. It is truly bizarre.

Is it only the first-mover myth that gets these companies such high regard in the industry? These are the kind of product issues you wouldn't expect with the kinds of technical interviews these companies use. Didn't they whiteboard inverting a red-black tree with anybody?

Not a tinder dev, but I am a mobile dev.

What Android device do you run?

What version of Android?

Is it stock Android from Google?

Stock firmware?

Not excusing the developers, but I want to point out that from my experience as a mobile developer, the sheer number of Android device configurations is a real problem to deal with, and often leads to lousy experiences on Android (along with Apps being a port of an IOS app, and teams outsourcing those ports to dev shops).

Nexus 5, latest Marshmallow. Vanilla Stock. I'm guessing it's triggered by a backend load thing, because it seems like it either just stops responding, or hangs when it can't download an image, and the app goes numb. If you see a spinner in Tinder, go to the square button and swipe it closed. It's simply a fragile app.

Good questions to ask, but Tinder on Android is truly terrible and the most unstable Android app I've ever used for any length of time.

It really casts the lie to the quality of work done by high-valuation companies.

Or that utility (in this case social/sexual) far outweighs things we devs might get hung up on. I keep using that app despite having it crash multiple times a day.

Well it's obviously the network effects that give it momentum, but sheesh. The existence of this app should obviate all future hiring threads.

Reducing dating app friction to "sign in with Facebook and choose a few pictures you already uploaded" is a double-edged sword.

20 comments * every item in the feed. I'm guessing their page size is something like 20 or 50 items. I could see dropping 400 objects out of memory as having an impact. Until you've actually done some performance testing it's hard to believe how strapped for resources some cheap Android phones really are.

Years ago we had OOM bugs killing a service. Turns out the developers were pulling the entire object graph and asset collection for every user displayed on a page.

Why not to use binary protocol? 20kb for just links and captions?

JSON is easier (faster) to parse on their client apps.

If you are building a new app, I'd really suggest checking out gRPC:

http://www.grpc.io/

The combination of an HTTP/2 stack w/ protocol buffers and libraries in everything you'd write a mobile app in is a powerful combo, and its a pretty high development velocity environment.

Just finished a fairly large pilot test with grpc. Summary: it's not production ready. It seems to be still moving rapidly and bugs pop up. I'd recommend holding back for awhile longer.

Also, instagram heavily relies on JS. There are not any protobuf parsers in js that perform anywhere near that of native json. The most feature complete implementation is two orders of magnitude slower.

Using the C++ bindings is faster, but is more difficult to fit into a typical build, can't work in a browser, and still an order of magnitude slower.

It's definitely something to keep an eye on, but, IMHO, not quite ready for something large scale.

How does it compare to flatbuffers [1]? Facebook apparently uses it, but their usage is from java [2].

[1] https://google.github.io/flatbuffers/ [2] https://code.facebook.com/posts/872547912839369/improving-fa...

gRPC is great, but Instagram's client apps are web-based (using React now), and consuming protobuf in JavaScript is unlikely to be faster (and definitely not as convenient) vs. JSON.

Given the limited resources of a lot of Android devices, an extra unmarshalling step would offset any wire-size reductions.

Just to clarify, do you mean that all of their mobile apps are on react-native now? Or just their web apps?

Extra unmarshaling? The string will either be parsed as JSON or some other thing.

Have you actually measured that? I've used a couple of different JSON libraries on Android and also some of the binary formats. In general the binary formats are more performant.

I definitely agree with you that JSON is easier to work with, but at the scale of Instagram I don't think that would be an issue.

Reply on news.ycombinator.com