Moving 6 Billion Messages Without Being Noticed
dt.deviantart.com
dt.deviantart.com
Percona toolkit has some useful stuff. Thanks for the link.
In this case our solution ended up being pretty much the same as pt-table-sync except that it made use of our distributed task queuing system so was MUCH faster than if we had used a single threaded perl script. (Up to a few hundred chunks of rows being checksummed/synced in parallel.) I didn't talk about this aspect directly in the article to keep it focused.
Not sure if it would be possible to use pt-table-sync in that way, certainly would have taken a lot more work to get a third-party perl script to distribute the work through our job queue.
What they really needed is ChronicDB http://chronicdb.com
The author of the 6b article kindly provided more information about the time it took for the migration:
http://news.ycombinator.com/item?id=3621561
A couple of limitations of the Percona tool are related to changing data in tables with ON DELETE and ON UPDATE foreign key constraints and possible lock-ups of tables. It is a bi-directional replication tool, so it has to deal with the master-master replication case and as such does not guarantee data consistency.
Still the tool is helpful in many cases and congrats to Percona for developing it. Replication safety is a non-trivial problem in general.
ChronicDB is a combination of a software engineer and a hosted database: You run your database in our datacenters -- http://chronicdb.com/how_it_works
Your database will be hosted in our datacenters -- http://chronicdb.com/features