HNHacker News
TopNewBestAskShowJobs

mikeyk

2,126 karma · joined January 6, 2010

Chief Product Officer @ Anthropic. Co-founder, former CTO of Instagram
submissionscomments
mikeyk··on Migrating From AWS to FB
Mike from IG here. VPC was barely a thing when we got on AWS (2010) and at the time not the default. I would definitely have done VPC from day 1 in hindsight, though.
mikeyk··on Instagram Django site admin
Mike from Instagram here. We've now locked it down more (the actual admin contents were always properly protected).

We're also part of Facebook's bug bounty whitehat program (https://www.facebook.com/whitehat/bounty/), if anyone comes across something in the future, we welcome responsible disclosure and pay out bounties through the program as well.

mikeyk··on React, a JavaScript library for building user interfaces
Yep--our Web products (including the logged-in Instagram.com) are built using React.
mikeyk··on Introducing Your Instagram Feed on the Web
Right now we support emoji if the browser does as well--for example, Safari on the Mac.
mikeyk··on Instagram Asking For Your Government Issued Photo IDs Now, Too
Hey all,

Mike Krieger (Instagram co-founder) here. Just wanted to offer some clarification since there's some speculation about the reason & scope for the verification mentioned in the article.

When we receive evidence of a violation of our site policies, we respond. This isn't a recent change, but the way we've run our community from the beginning. In some specific cases, for verification purposes, we request that people upload a government issued ID in response. This is the case for a very small percentage of accounts, and doesn't affect most Instagram users. The ID is used only for account verification, and not retained permanently.

Hope that helps clarify things a bit.

mikeyk··on Handling Growth with Postgres
We're very happy with it--given the constraints it's working well. The biggest drawback with our sharding/ID scheme is that it's harder to split off a single user if you need to special case.
mikeyk··on Airbnb Tech Talk: Instagram's Mike Krieger on scalability, android launch
Very excited to give this talk--if any of you have particular things you'd like to see me address let me know and I'll see if I can fit it in.
mikeyk··on Keeping Instagram up with over a million new users in twelve hours
You can hypothetically use something like hearbeatd to do it; we run every Redis master with an attached slave and manually failover for now.

For a small team like ours, we prefer solutions that are easy to reason about and get back into a healthy state (it would take one server deploy to point all appservers at new Redis master), rather than fully automated failover and the "fun" split-brain issues that ensue. Of course that may change as we build out our Ops team, etc.

mikeyk··on Keeping Instagram up with over a million new users in twelve hours
We've had to split munin across three masters (by machine role) because the graphing job was just locking on IO. Munin 2.0 moved over to all-dynamic CGI graphing, but I haven't gotten the chance to play with it yet.
mikeyk··on Keeping Instagram up with over a million new users in twelve hours
They are, but we software-RAID our EBS drives to get better write throughput, and we put the Write-Ahead Logs (WALs) on a different RAID from the main database, so when you have both of those going on, you need something else to atomically snapshot our PG databases.
mikeyk··on Keeping Instagram up with over a million new users in twelve hours
Those are really useful numbers--I think a lot of it can be chalked up to virtualization, but we should definitely explore more around IRQ pinning for queues. Any good starting points / reading, are you mostly using taskset?
mikeyk··on Keeping Instagram up with over a million new users in twelve hours
Yes, there is--we take all of our snapshots from a slave, and we stop the slave before taking a snapshot, then XFS-freeze all drives, then take the snapshot, to ensure it's consistent.
mikeyk··on Keeping Instagram up with over a million new users in twelve hours
To clarify: we did do a fair amount of capacity planning and elastic ramp-up/pre-allocating of our infrastructure, but no prediction is perfect, so the blog was about diagnosing and addressing issues as they crop up under unprecedented load.
mikeyk··on Keeping Instagram up with over a million new users in twelve hours
We have a variety of Redis machines, some of them are in a consistent hash ring (the ones we're using for caching); some are using modulo-based hashing (the ones where losing data on adding more machines isn't an option), and some are just single-node installs.
mikeyk··on Keeping Instagram up with over a million new users in twelve hours
We use Ubuntu, running Natty Narwhal. Every Ubuntu version before that had some (unique) bugs that would bite us under load. Natty has been by far the most stable.
mikeyk··on Keeping Instagram up with over a million new users in twelve hours
NewRelic's product screenshots look really nice, but 1) the hosted nature and 2) the price were turnoffs. We might revisit it at some point, though--there are a few features in there that would be awesome to have.
mikeyk··on Keeping Instagram up with over a million new users in twelve hours
Thanks! This was my first node.js project, would love any feedback on it.
mikeyk··on What Powers Instagram: Hundreds of Instances, Dozens of Technologies
Hey, author here. To clarify, our uploads go to our servers first (where we resize for thumbnails, etc) then go to S3.
mikeyk··on Storing hundreds of millions of simple key-value pairs in Redis
Yep, ~30 inserts per second go in, so it's not static.
mikeyk··on Storing hundreds of millions of simple key-value pairs in Redis
Honestly, it was mostly the concision that struck me at the time. Looking at the zipmap source now, I think it's more or less where my code tends to end up in terms of comments--some motivation / higher-level comments at the top, and most functions have a comment before, hopefully with some explanation of any edge/NULL cases as well. Any open-source projects you'd point to as good examples of what looks readable to you? Always trying to improve my own coding habits as well.
mikeyk··on Storing hundreds of millions of simple key-value pairs in Redis
Back when we were getting started with Redis, the readability / concise nature of the project was one of the things that most excited me about it (here's what it looked like around then: https://github.com/antirez/redis/tree/0b420168b485d0a9c4b66d...)
mikeyk··on Simplifying EC2 SSH connections
We run several dozen instances, so managing that list manually became cumbersome. Being able to create an instance, and moments later have that instance easily SSH-able by anyone on the team without requiring an additional commit & pull has been useful for us.
mikeyk··on Sharding and IDs at Instagram
We love Redis at Instagram (it powers a lot of our systems...more write-ups on these soon), and considered it for our ID generation, but it would have introduced a single point of failure, unless we split the load between several Redis instances, at which point it would be hard to make the IDs time-sortable. Also, most of our Redis systems are durable within a minute (we write to disk on a slave every minute), but if we were to lose the master and slave simultaneously (imagine an EC2 network issue or such), then it would be hard to know what the last known 'good' ID was--not an insurmountable problem, but one more moving part to worry about. That said, using Redis with something like the upcoming Redis Cluster could be a good choice, though.
mikeyk··on Sharding and IDs at Instagram
Hey Ken, (author here)

There's a 1:1 mapping between the user's hash modulo the number of shards, and the table they're writing to. So, if we had 1000 logical shards, we have 1000 schema/tablespaces, with the same tables in each. And the database's own 'nextval()' feature makes sure never have the same ID twice. Hope that clarified things.

mikeyk··on Sharding and IDs at Instagram
I used to work at Meebo, which hosts a large CouchDB system , but when it came time to choose a solution for Instagram we went with PGSQL. There was a lot I really liked about Couch, especially for the analytics system we built at Meebo (where the map/reduce views worked great), but I wouldn't classify it as a low-Ops-burden technology--like any newer db solution, there are some rough edges and more 'unknowns' at scale.

Also, we'd still have to write the middleware to assign data to shards and fetch data from shards in our system (unless we used something like BigCouch), so having a more tried-and-tested solution that we already understood well was more appealing.

mikeyk··on Sharding and IDs at Instagram
Thanks for the info, I updated the post to mention that the write bottleneck isn't an issue at Flickr.
mikeyk··on Sharding and IDs at Instagram
Valid questions!

We're on EC2, which has its set of limitations but means we can run a 10 million + user system with two-and-a-half engineers (and no ops team / overhead). So while we hear about more and more folks using SSDs in their DBs, it's not an option in our near-term future.

For SQL vs HBase/Haystack, we don't really have to worry about the photo storage itself, since S3 handles all of it. The data we shard out is more suited to an RDBMS, and since we're way more familiar with that world than with HBase and similar, it was the choice that let us make the most progress in a short time with a small team. Hope that's a helpful description of how we thought about it.

mikeyk··on Sharding and IDs at Instagram
Yep--definitely something to watch out for. At worst, though, you'd have a duplicate key when trying to insert, and can re-try without the risk of having a duplicate ID floating around your system.
mikeyk··on Sharding and IDs at Instagram
We considered the composite key, but we often have to store keys in other systems like Redis, where having a single 64-bit integer makes it more portable and stored compactly (Redis, for example, has an optimization when storing integer values vs string values in its lists). Valeu pelo comentário!
mikeyk··on Sharding and IDs at Instagram
We set-up the schemas ahead of time, and then each lookup is (with, say, 1000 schemas):

  user_id % 1000 -> schema ID
  schema ID -> database ID
then select FROM schemaID.tablename etc on that particular database.
← PreviousPage 2 of 3Next →