We have a worker consuming S3 events and updating the index. On a related note, we have experimented with Lambda to do something similar; AWS has done a fantastic job integrating their products :)
That is, what happens if a db write fails? How are you handling concurrent updates? Is there a reconcile process that runs periodically?
(those details would be an awesome engineering blog post :) )
Edited to add:
Any reason that AWS Lambda didn't work out? Was it due to the public endpoint requirements?
Double Edited to add:
I totally geek out on composing AWS Services, and this is fascinating to me.
We are eagerly waiting for Lambda VPC support!
I'm happy you're interested in this sort of stuff; shoot me an email, let's chat :) michael@segment.com