Mike from IG here. VPC was barely a thing when we got on AWS (2010) and at the time not the default. I would definitely have done VPC from day 1 in hindsight, though.
Mike from Instagram here. We've now locked it down more (the actual admin contents were always properly protected).
We're also part of Facebook's bug bounty whitehat program (https://www.facebook.com/whitehat/bounty/), if anyone comes across something in the future, we welcome responsible disclosure and pay out bounties through the program as well.
Mike Krieger (Instagram co-founder) here. Just wanted to offer some clarification since there's some speculation about the reason & scope for the verification mentioned in the article.
When we receive evidence of a violation of our site policies, we respond. This isn't a recent change, but the way we've run our community from the beginning. In some specific cases, for verification purposes, we request that people upload a government issued ID in response. This is the case for a very small percentage of accounts, and doesn't affect most Instagram users. The ID is used only for account verification, and not retained permanently.
We're very happy with it--given the constraints it's working well. The biggest drawback with our sharding/ID scheme is that it's harder to split off a single user if you need to special case.
You can hypothetically use something like hearbeatd to do it; we run every Redis master with an attached slave and manually failover for now.
For a small team like ours, we prefer solutions that are easy to reason about and get back into a healthy state (it would take one server deploy to point all appservers at new Redis master), rather than fully automated failover and the "fun" split-brain issues that ensue. Of course that may change as we build out our Ops team, etc.
We've had to split munin across three masters (by machine role) because the graphing job was just locking on IO. Munin 2.0 moved over to all-dynamic CGI graphing, but I haven't gotten the chance to play with it yet.
They are, but we software-RAID our EBS drives to get better write throughput, and we put the Write-Ahead Logs (WALs) on a different RAID from the main database, so when you have both of those going on, you need something else to atomically snapshot our PG databases.
Those are really useful numbers--I think a lot of it can be chalked up to virtualization, but we should definitely explore more around IRQ pinning for queues. Any good starting points / reading, are you mostly using taskset?
Yes, there is--we take all of our snapshots from a slave, and we stop the slave before taking a snapshot, then XFS-freeze all drives, then take the snapshot, to ensure it's consistent.
To clarify: we did do a fair amount of capacity planning and elastic ramp-up/pre-allocating of our infrastructure, but no prediction is perfect, so the blog was about diagnosing and addressing issues as they crop up under unprecedented load.
We have a variety of Redis machines, some of them are in a consistent hash ring (the ones we're using for caching); some are using modulo-based hashing (the ones where losing data on adding more machines isn't an option), and some are just single-node installs.
We use Ubuntu, running Natty Narwhal. Every Ubuntu version before that had some (unique) bugs that would bite us under load. Natty has been by far the most stable.
NewRelic's product screenshots look really nice, but 1) the hosted nature and 2) the price were turnoffs. We might revisit it at some point, though--there are a few features in there that would be awesome to have.
Honestly, it was mostly the concision that struck me at the time. Looking at the zipmap source now, I think it's more or less where my code tends to end up in terms of comments--some motivation / higher-level comments at the top, and most functions have a comment before, hopefully with some explanation of any edge/NULL cases as well. Any open-source projects you'd point to as good examples of what looks readable to you? Always trying to improve my own coding habits as well.
We run several dozen instances, so managing that list manually became cumbersome. Being able to create an instance, and moments later have that instance easily SSH-able by anyone on the team without requiring an additional commit & pull has been useful for us.
We love Redis at Instagram (it powers a lot of our systems...more write-ups on these soon), and considered it for our ID generation, but it would have introduced a single point of failure, unless we split the load between several Redis instances, at which point it would be hard to make the IDs time-sortable. Also, most of our Redis systems are durable within a minute (we write to disk on a slave every minute), but if we were to lose the master and slave simultaneously (imagine an EC2 network issue or such), then it would be hard to know what the last known 'good' ID was--not an insurmountable problem, but one more moving part to worry about. That said, using Redis with something like the upcoming Redis Cluster could be a good choice, though.
There's a 1:1 mapping between the user's hash modulo the number of shards, and the table they're writing to. So, if we had 1000 logical shards, we have 1000 schema/tablespaces, with the same tables in each. And the database's own 'nextval()' feature makes sure never have the same ID twice. Hope that clarified things.
I used to work at Meebo, which hosts a large CouchDB system , but when it came time to choose a solution for Instagram we went with PGSQL. There was a lot I really liked about Couch, especially for the analytics system we built at Meebo (where the map/reduce views worked great), but I wouldn't classify it as a low-Ops-burden technology--like any newer db solution, there are some rough edges and more 'unknowns' at scale.
Also, we'd still have to write the middleware to assign data to shards and fetch data from shards in our system (unless we used something like BigCouch), so having a more tried-and-tested solution that we already understood well was more appealing.
We're on EC2, which has its set of limitations but means we can run a 10 million + user system with two-and-a-half engineers (and no ops team / overhead). So while we hear about more and more folks using SSDs in their DBs, it's not an option in our near-term future.
For SQL vs HBase/Haystack, we don't really have to worry about the photo storage itself, since S3 handles all of it. The data we shard out is more suited to an RDBMS, and since we're way more familiar with that world than with HBase and similar, it was the choice that let us make the most progress in a short time with a small team. Hope that's a helpful description of how we thought about it.
Yep--definitely something to watch out for. At worst, though, you'd have a duplicate key when trying to insert, and can re-try without the risk of having a duplicate ID floating around your system.
We considered the composite key, but we often have to store keys in other systems like Redis, where having a single 64-bit integer makes it more portable and stored compactly (Redis, for example, has an optimization when storing integer values vs string values in its lists). Valeu pelo comentário!