Amazon Aurora with PostgreSQL Compatibility Supports Serverless | Hacker News Reader

Amazon Aurora with PostgreSQL Compatibility Supports Serverless

aws.amazon.com

88 pointskesor30 comments

For clarity, there is a significant warm up time for the DB to load your data when you cold-start, so this is sadly not suitable for use with serverless lambda functions in a web request cycle.

A good use case for this would be batch processing on demand e.g. a background job that's infrequently run when records are uploaded somehow. You save the cost of keeping a big DB running in between jobs and the start up time doesn't matter.

What kind of time are we talking about, 10 seconds or 3-4 minutes?

Or very short, but everything will be painfully slow?

On the order of tens of seconds, for a cold start from zero. Enough time to load your RAM cache from hibernation and boot up OS and Postgres.

Is it really? Been trying out Aurora MySQL for our staging db, and its closer to 1 min+ for a cold start.

But isn’t the variance on this completely useless? It’ll depend entirely on the size of your cache, the region you’re in, the traffic at that time and rack you’re lucky enough to be on. Will also depend on AWS configs, which can change daily.

But what does it mean to be warm in this case? Last called less than a minute ago? 10 minutes?

Is their a hook on spin down? You could send a small read only cache to a S3 SQLite file so lambdas will have a failover on long reads.

Just don't let it spin all the way down. Scaling up and down is better.

Also note that you don’t necessarily have to scale down all the way to zero. Also works for workloads that are very spiky, like breaking news or weekend or work hours heavy loads.

If pricing is similar to other serverless offerings at AWS, then it will cost much more to run it constantly at low rates than to just run a small instance all the time. But maybe the much faster scale up would be desirable in some cases.

AWS RDS for PostgresSQL supports many popular extensions [1]. I wonder if these are activated here or will be in time. One of pg's unfair advantage is its ecosystem.

I have user-triggered batch processing jobs that need PostGIS; Aurora Serverless for pg would be a great fit.

[1] https://docs.aws.amazon.com/fr_fr/AmazonRDS/latest/UserGuide...

I haven't had a chance to try it out myself yet, but the Aurora FAQ[0] seems confirm your wish:

Aurora database engine is designed to be wire-compatible with PostgreSQL 9.6 and 10, and supports the same set of PostgreSQL extensions that are supported with RDS for PostgreSQL 9.6 and 10

For reference, full list of PostgeSQL RDS extensions in the documentation[1]

[0] https://aws.amazon.com/rds/aurora/faqs/ [1] https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_...

Oh, that's great!

Thanks

Still waiting for AWS RDS for PostgresSQL to support the TimescaleDB extension, 2 years and counting:

https://github.com/timescale/timescaledb/issues/65

Although I know AWS is not optimizing for this use case, I see a lot of potential with serverless databases for things like open source, home brew style applications where people want to own and control their data but don't want to pay for a monthly service.

IoT devices could communicate asynchronously and store important data using just llamdba functions and serverless db instances.

The first thought that came to mind was something like the bitwarden open source server. Currently, you can run a nano instance in the cloud that stays on forever but it needs to be maintained and it might go down occasionally. Additionally, it will cost you a few bucks a month assuming you need more than the single nano allowed by the free tier.

With serverless functions and a serverless database you could have a central store without needing to run a full server. Cold startup wouldn't be an issue since you only need to sync to the server occasionally.

This is on HN like just now https://news.ycombinator.com/item?id=20398353

Aurora with Postgres is amazing. I don't know if I want to go back to Redshift after using it. Pair it with a nice instance of PGADMIN4 and enjoy.

They're pretty different databases with different use cases in mind. If you don't really have big data you can get away with a row oriented database. If you do have big data for reals - well then you almost certainly have to use a column oriented store.

I can't tell if you're being sarcastic, or if your tastes in software are extremely distinct from mine and most--comparing Aurora to Redshift? Pgadmin4, really??

I've cut costs by an order of magnitude by changing back to old school dedicated servers. ( using the same data center that aws/google uses, equinix ) IOPS on aws is ridiculous expensive

1) AWS and GCP have their own datacenters with their own custom switches and routers, etc.

2) Sure, as a DBA, I prefer dedicated servers with SSD. If you don't have a DBA, then RDS/Aurora are ok too.

For production, ensure you have a replica either way. MySQL 8 has multi-master now (or use Percona Server/Galera), so you can easily enable HA, and I'm sure Postgres does too.

How well would this work with something like airflow ? If I have jobs that run only end of day for an hour on airflow, I pay just for an hour of the database ?

Depends on how you've configured airflow. It polls its dag directory constantly, and may do some db queries for each poll (don't quote me on this, I definitely don't know everything about the internals of airflow). From what I remember you can reduce the polling period in config

Yeah I’ll need to do some research, if I can only poll the db for just the time I need this will be a huge cost saving for me. I think the db only gets queried if a dag actually runs but I’m not 100% sure.

Has anybody here used Amazona Aurora serverless?

Is it a viable alternative for big RDS instances?

A company that I work for was in the beta of serverless for MySQL, and we currently use it for both dev and production workloads. We're a startup, so it definitely simplifies devops for us, but personally I think you're better off just running a large RDS instance.

The main problem with it that we have run into is that if there are active transactions in the database, it can't find a "scale point". So when our application gets hit with thousands of users at a time and they are all generating database transactions, it can't scale up.

It's definitely been good for us in development though, and I would highly recommend it for that.

They do allow you to drop connections with forced scale points if that's a tradeoff your application can make https://aws.amazon.com/about-aws/whats-new/2019/04/amazon_au...

From a cost perspective, when I last checked, it was quite a bit more expensive to go serverless unless you have very intermittent demand or have very spikey and unpredictable traffic.

Is this still capped at data of 64TB?

Smaller, 32 TiB.

https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide...

Reply on news.ycombinator.com