Hacker News Reader

Comment by Agebor | Hacker News Reader

Not sure what do you mean about scaling storage layer separately from the compute layer. It's how it's usually done currently, with the disk being the storage layer.

As long as there are machines/containers to be started, there will always be some latency, though we can expect it to improve in the next years.

AWS now has the Aurora Serverless, though I don't have any real-life experience with it's latency yet (VPC requirement is a downer for now). https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide...

View on HN

> Not sure what do you mean about scaling storage layer separately from the compute layer. It's how it's usually done currently, with the disk being the storage layer.

Yeah, I meant that only. DynamoDB and Aurora Serverless are examples where they are decoupled and can be scaled independently. So, I wanted to know about more products like them but ones that scale with load really well.

> AWS now has the Aurora Serverless, though I don't have any real-life experience with it's latency yet (VPC requirement is a downer for now).

I have tried Aurora Serverless and I like it but it also plagued by the provisioning problems for now. The lack of options to customize database parameters was kind of a downer. Also for some weird reason they only support MySQL in t type instances.

> (VPC requirement is a downer for now)

BTW, you have an use case where you need to use DB from outside VPC and you can't do peering?

I don't understand what you're looking for here. A network file system? Every database decouples storage from "compute."

Yeah, but not all of them allow you to scale them independently without a downtime.

For example in traditional MySQL setups to scale up the compute capacity your server has to be unavailable for some time.

I think people way overestimate the business effects of maintenance windows.

Disagree completely. Services that you can mostly depend on to never go down, and scale as far as you will ever want, are a huge relief. Like S3.

We should ask for the same thing with rdbms's.

The overestimation is in whether a company should be using s3 as an operational standard.

If you take them once a quarter/year, they are fine. But if you have load patterns where you want to pay for what you use and not over provision the DB significantly then you can't rely on maintenance windows. I think we will get there eventually with even RDBMS getting solutions like Aurora Serverless but right now most of them don't work well.

Google Cloud's BigTable is like this. I don't remember how long scaling up took but instead of telling it how many "IOPS" to provision, you tell BigTable how many compute nodes you want. Once you hit ~80% on a node, the latency would rise, until you added another compute node. Since storage/compute was decoupled you wouldn't have to "wait" for a rebalance either.

> BTW, you have an use case where you need to use DB from outside VPC and you can't do peering?

Lambda in VPC have unacceptable cold-start for interactive applications. This is partly why DynamoDB use is so popular with Lambda, it does not have to be in VPC.

Reply on news.ycombinator.com