Amazon DynamoDB On Demand | Hacker News Reader

Amazon DynamoDB On Demand

aws.amazon.com

105 pointsabd1223 comments

Finally! One of the worst things as a startup was trying to capacity plan my DynamoDB when I had no idea what the useage pattern would be. Now I can not worry about it while I get data for future capacity planning.

There was already had DynamoDB Auto Scaling to achieve that. Main benefit of DynamoDB On-Demand from my point of view is:

- easier setup (a single setting instead of configuring auto scaling rules and policies for the table and all its indicies)

- possibly cheaper (depends on the not yet published details of the per-request pricing)

Auto scaling lagged though. And you still had to guess at the base level. They mentioned in the keynote that this is a big improvement over auto scaling.

I am puzzled by this statement: Tables using on-demand mode support all DynamoDB features (such as encryption at rest, point-in-time recovery, global tables, and so on) with the exception of auto scaling, which is not applicable with this mode. Source: https://aws.amazon.com/blogs/aws/amazon-dynamodb-on-demand-n...

Does it mean auto scaling is no longer needed, or it won't happen in this mode (some hidden fixed capacity?)

With DynamoDB On-Demand AWS handles scaling transparently for the customer. So they'll definitely scale, but as a customer you shouldn't notice it. I'm quite curious how that works out in practice with rapidly changing request rates.

Thanks!

Why I've been attracted to azure table storage and google firestore.

Capacity planning is hard.

I think it is about time... Azure Storage Tables have been that way pretty much from the start, comparatively.

Doesn't seems to be attractive price-wise.

Quick math (us-east-1 North Virginia, reads, strong consistency):

1. Provisioned:

$0.00013 per RCU

1 RCU provides 3,600 reads per hour

0.00013 * 1000000 / 3600 ~= $0.036 per mln reads

https://aws.amazon.com/dynamodb/pricing/provisioned/

2. On-demand:

$0.25 per million read request units

https://aws.amazon.com/dynamodb/pricing/on-demand/

So AWS on-demand is more expensive if your read utilisation is higher than 14.4%. Not that exciting for any serious use-case.

Plus you can reserve DynamoDB capacity for a year and save ~56%: (30 + 0.0025 * 24 * 365) / (100* 0.00013 * 365 * 24 )

That calculation only works if you can spread your workload over ten minutes in the hour. If you want to get it done in ten seconds, then you don’t have any more to do for the next hour, then on-demand is much better.

Love that this feature finally released. I was a dev manager on DynamoDB years ago when we first started working on this. I left three years ago and had almost given up hope that this feature would see the light of day.

The great thing about it is that it helps with spiky and sporadic workloads. In my current work I have a bunch of unit tests that hit a set of dynamodb tables. I have those tables provisioned at a 5 RCU and 5 WCU each. Multiply this by ten tables and I'm paying $25 per month. In the grand scheme of costs, this isn't much, but it's annoying and adds up. (One can argue that unit tests should be hitting a mock or a dynamodb local. That's true, but you also want some tests that actually hit the service.)

On demand can be great for sporadic tests, for prototypes, for small personal projects, and for just messing around.

There are also spiky non-customer-facing production scenarios that can benefit from On demand.

Am I correct in thinking Google Cloud Datastore has this already? What are the biggest remaining differences between DynamoDB vs Datastore?

DynamoDB has a simpler key/value data model like BigTable. Cloud Datastore is a document-store with a richer data model (built on top of BigTable, or at least it was until the latest version with Firestore).

It looks like you will be charged more for on-demand but I don't see a list of the rates yet, anyone find them?

The pricing page is available at https://aws.amazon.com/dynamodb/pricing/on-demand/ , but it looks like it's not yet updated with the per-request rates. The blog post mentions rates in us-east-1: "in the US East (N. Virginia) region, you are charged $1.25 per million write requests units and $0.25 per million read request units, plus the usual data storage costs."

My math:

Provisioned: $0.00065 per WCU. One WCU is 1 write per second for an hour = 3600 writes. So 277 WCU is 1 million writes in an hour. Cost is $0.18 per million write requests. (If you reserve this for a year it's $0.08 per million write requests.)

We generally run our tables with autoscaling set to 70% of capacity, so at $0.18 per million write requests our effective cost is $0.25 per million write requests. A good deal cheaper than on demand.

We have a few tables where we run closer to 30% of target capacity because of hot shards. There our effective cost is $0.60 per million write requests, which is still cheaper than on demand (assuming on demand doesn't have the hot shard problem).

I'm having trouble understanding when this would be beneficial.

This is actually meant for companies like ours. We have a very spiky workload (need massive throughput needs and then nothing happens for rest of the day). The current auto-scaling doesn't work since it takes time to scale up and you are limited to only few of those changes per time period. We ended up setting the throughput high to avoid that. Also people in company set up these high throughput on dynamodb and we end up paying thousands of dollars more before this is discovered. I know we can write lambda's to automate the detection and all of that but that is too much work. This feature is ideal and will drive our costs down a lot.

I'm considering using this for our staging tables since 90% of the time, we'd want to scale to 0 instead of 1. Doesn't do much for cost savings, but it does make it easier to manage.

One use case where it would be beneficial is for tables that have batch-style patterns, where you (at least in my experience) need to provision a good enough baseline before autoscaling kicks in.

In some cases I've seen, autoscaling can manage going from ~500 to 10k WCU without heavy throttling (as the burst capacity can handle the five~ten minutes before autoscaling kicks in), but not with a smaller baseline. On tables with hot shards, the baseline usually has to be higher.

This would be beneficial for companies who have not yet figured out the math like you have. Also for companies who have 5-6X spikes in a non predictable fashion. Also, even for predictable spikes, there is a non-trivial lag for autoscaling.

Probably also useful for multi-tenant apps like web analytics. One cannot possibly predict the workload.

If you only pay for what you use then that seems reasonable.

It's nice to see that DynamoDB becomes truly serverless (although it lived under the serverless umbrella for a while) with this enhancement.

But I feel the on-demand mode should be best accompanied with the provisioned mode, which is the hybrid mode. So instead of using the laggy autoscaling to handle the burst traffic, using the on-demand capacity to serve that.

Disclosure: I work for Alibaba Cloud and use Alibaba Cloud Table Store which provides the hybrid mode ( https://www.alibabacloud.com/help/doc-detail/27291.htm )

Reply on news.ycombinator.com