HNHacker News
TopNewBestAskShowJobs

mikeyk

2,126 karma · joined January 6, 2010

Chief Product Officer @ Anthropic. Co-founder, former CTO of Instagram
submissionscomments
mikeyk··on Sharding and IDs at Instagram
Thanks for the comment! We don't need the IDs to be exactly sortable, only roughly sortable within a second or so. As long as the clock doesn't move backwards on any given machine (we use ntpd in its gradual-adjustment mode), the IDs are unique.

The way we move shards is to use PostgreSQL's built-in streaming replication to create an exact, in-sync copy of a set of tablespaces, then 'fail over' to a new machine and start reading/writing to a subset of those tablespaces (this is similar to how Facebook describes their shard-moving process).

mikeyk··on Sharding and IDs at Instagram
Right now, it's a lookup dict in our Django app--which involves brief downtime just to update the shard map when moving the data (more on this in another post). We hash on user ID (in most cases) and then look the shard # in the dict, then look up the shard # in a logical-to-physical dict.

By the way we use & love Sentry!

mikeyk··on Sharding and IDs at Instagram
(author here) I think this should also be possible with MySQL's stored functions, but one huge benefit to PostgreSQL is the schema/tablespace feature, since it means all our logical shards all live inside one database (you could do something similar with MySQL, but it would mean multiple databases or prefixing table names with the shard ID).
mikeyk··on Django SQL Sampling
Pgfouine is awesome, but one thing that's appealing about this one is the traceback showing where the query occurs--not always obvious given the Django ORM.
mikeyk··on Launching your iPhone & iPad App With Amazon Web Services
The DB write spikes are very likely to be EBS glitches, we've seen them do that frequently.

The crazy CPU issue might be related to the EC2 bugs in the Lucid Lynx kernel...Maverick is also problematic but Natty Narwhal has fixes for many of the weird Ubuntu-on-EC2 issues

mikeyk··on "I trust Redis-on-disk every day less." --Salvatore
You can run a slave Redis replica very easily, and save db snapshots from that without impacting the master. When Salvatore says "redis on disk", he's referring to attempts to mix in-memory & on-disk portions of the data, rather than the background saving feature of Redis.
mikeyk··on EC2 Users Should be Cautious When Booting Ubuntu 10.04 AMIs
Seconded. We had our 10.04 machines regularly, and become totally unresponsive. Then, we had even worse luck with Maverick--starting with the issue where it didn't recognize half the RAM on the 68GB instance type. Natty seems to be holding up the best so far.
mikeyk··on How Fucked is Muni
Thanks for this! My dream Muni app would show the average time it's currently taking to get through the Market underground section (where I see most of the bottlenecks, esp. on the N). This is a good proxy in the meantime :)
mikeyk··on Robert Haas: MySQL vs. PostgreSQL, Part 2: VACUUM vs. Purge
Plus, PostgreSQL built in master-slave streaming replication in release 9, so scaling out reads is even less of an issue.
mikeyk··on MySQL in the cloud at Airbnb
Would have loved to have seen some actual perf numbers, especially in terms of how much of a cost the Multi-AZ writes entail. We're doing Postgres on EC2, which means RDS doesn't quite help, but Postgres 9's hot standby/streaming replication at least gets a bit closer (and could be easily extended to a Multi-AZ set-up if one of our hot standbys was in a different AZ).
mikeyk··on Screenshot: "A Fire Upon the Deep" sequel (2011)
If you're a Vinge fan, the annotated version of A Fire Upon the Deep is a great find (available on Kindle); it has his emacs notes (linked in-text) and you can watch the characters develop, his editor suggest improvements, etc. Looks like his system hasn't changed all that much, either.
mikeyk··on Dear Twitter: Stop screwing over your developers
Nope; xauth reduces the number of steps a user has to go through for your service to obtain an access token for them (you can prompt them for their username and password rather than launch a web view pointing at Twitter.com). You still need to ship your consumer key with the app, even with Xauth.
mikeyk··on Review my startup: Presence
I read this last night and this morning and still can't quite figure out what your app does. Focus your message, and provide some illustrations/scenarios of how this would matter to someone real, and it will help the communication aspect.
mikeyk··on Cappuccino GitHub Issues App Update
Nice! 2 questions:

  1. Are the keyboard shortcuts documented anywhere?
  2. Is there a way to reorder the issues in the main view?
mikeyk··on Ask HN: Will there ever be an eHarmony for finding friends?
Meet Gatsby does this with foursquare (http://meetgatsby.com/)
mikeyk··on Crime Visualization of San Francisco
I made CrimeDeskSF (http://itunes.apple.com/us/app/crimedesksf/id339706696?mt=8) to visualize some of this data on the iPhone, including a mostly-gimmicky-but-kinda-fun augmented reality view (drug bust 100ft away!). I'm using the datasf.org datasets for mine (not sure how they compare to the crimespotting ones, or if they're the same).
mikeyk··on Mailplane preview integrates Rapportive
a hearty endorsement here for Mailroom: it takes exactly what I love about Mailplane on the Mac (switching between accounts; better at-a-glance indication of new messages; letting Gmail be Gmail instead of shoving it through IMAP and suffering through the sync/search/etc issues that will inevitably happen).
mikeyk··on Why Textmate 2.0 is not Developed in the Open
Re: the full-project rescan (a nightmare when using TextMate with ExpanDrive), ReMate is your friend: http://ciaranwal.sh/remate/
← PreviousPage 3 of 3