As each instance starts, it boots its AMI. That image includes a snapshot of the checked-out eBook repository as of the date the image was created. Each gerbil, as it starts, updates this cache to get the latest version. This involves checking out perhaps 500Mb of data per instance, as new books will have been added and existing books updated since the instance was frozen. As a result, it will often take 30–45 minutes to start 10 instances.
Am I off base here thinking that a fairly obvious solution to address the 30-45 minute cold start is to NOT pull the entire library of book templates from svn at startup? Rather, why not simply update just the template(s) currently stacked up in the job queue? So instead of a 500MB pull your looking at maybe 1 or 2 before you can start processing a job in the queue.
I wonder if this "challenge" is an existing problem or something they've already addressed. Has relying only on "on demand" cache updating been tried already (seems like it would have been since it seems like an obvious possible solution) if so what was the result?