Hacker News Reader

Considering their "Continuous Protection" feature, could they be streaming a write log out to S3? I think that way they could use the backup as the sole guarantee of durability, and use instance storage for the database.

EDIT: also, I'm not sure if this is generally known elsewhere, but might they not be on ec2, they are advertising 1.7 GB cache, which is the same as a small instance, except without any room for the operating system or postgres itself.

View on HN

You're exactly right about log streaming to S3. We open sourced the tool that we use to perform that: https://github.com/heroku/wal-e

S3 doesn't support continual logging - you need to decide on a file size up-front. Are you creating a new object for each log entry, or is there a window in which commits can be lost before the log gets flushed out, or do you have some other workaround...?

There's a window of a couple minutes in which data could be lost (in the worst case scenario).

So it's postgres log shipping, with S3 as the target?

Yup.

So would I need to (learn and) use PG9's replication with this? Or is one ephemeral-stored database plus shipped logs sufficient, assuming I can live with some downtime if the instance disappears?

As an experienced techie but a PG novice, what do I need to go learn about PG recovery procedures to safely use this?

Reply on news.ycombinator.com