Hacker News Reader

Exactly same case here - we're using Athena (packaged presto from AWS) and have a Postgres instance w/ identical data in a different data model. Presto is mindblowingly good at the low latency queries and scales well to anything nontrivial. It seems opening files for reading is the biggest source of delay in presto! Postgres is not handling the scale well, but it was a nice experiment.

View on HN

Your Athena data set are immutable S3 blobs of data however, while Postgres reads data that could be modified while it's reading them, concurrently.

Postgres has to deal with a transaction modifying 100,000 rows in a table, needing that transaction to then see its changes but then aborting, leaving the data exactly as it was before. Athena/Presto don't have to worry about any transactions, table files bloated by changes that need vacuuming, etc. etc.

So I don't think it's fair to claim PG does not scale, when you are comparing a transactional and analytical database.

That's rather the point though. The people saying "YAGNI, just use Postgres" aren't, a priori, any more correct than people saying "You definitely need Hadoop if you're gonna scale!"

You need to analyze our own use case critically, understand the tradeoffs of different tools and choose pragmatically.

Reply on news.ycombinator.com