Hacker News Reader

You're exactly right about log streaming to S3. We open sourced the tool that we use to perform that: https://github.com/heroku/wal-e

View on HN

S3 doesn't support continual logging - you need to decide on a file size up-front. Are you creating a new object for each log entry, or is there a window in which commits can be lost before the log gets flushed out, or do you have some other workaround...?

There's a window of a couple minutes in which data could be lost (in the worst case scenario).

So it's postgres log shipping, with S3 as the target?

Yup.

So would I need to (learn and) use PG9's replication with this? Or is one ephemeral-stored database plus shipped logs sufficient, assuming I can live with some downtime if the instance disappears?

As an experienced techie but a PG novice, what do I need to go learn about PG recovery procedures to safely use this?

Reply on news.ycombinator.com