abd12

I just answered this on Twitter, but I think there are two instances where it's a no-brainer to use DynamoDB:

- High-scale situations where you're worried about performance of a relational database, particularly joins, as it scales.

- If you're using serverless compute (e.g. AWS Lambda or AppSync) where traditional databases don't fit well with the connection model.

That said, you can use DynamoDB for almost every OLTP application. It's just more a matter of personal preference as to whether you want to use a relational database or something like DynamoDB. I pick DynamoDB every time b/c I understand how to use it and like the other benefits (billing model, permissions model, performance characteristics), but I won't say you're wrong if you don't choose it in these other situations.

It's different than a relational database in that you need to model your data to your patterns, rather than model your data and then handle your patterns.

Once you learn the principles, it really is like clockwork. It changes your process, but you implement the same process every time.

Honestly, I think part of the problem is that there's a lot of bad NoSQL content out there. A little standardization of process in this space will go a long way, IMO :)

Thank you! Glad you found DynamoDBGuide.com helpful :)

Yep, I do talk about aggregations in the book. One strategy that I've discussed is available in a blog post here[0] and involves using DynamoDB Transactions to handle aggregates.

If you're looking for large-scale aggregates for analytics (e.g. "What are my top-selling items last month?"), I have an Analytics supplement in the Plus package that includes notes on different patterns for analytics w/ DynamoDB.

Let me know if that helps!

[0] - https://www.alexdebrie.com/posts/dynamodb-transactions/#hand...

We've all been there! :)

Thanks for your support. I really appreciate it. Don't hesitate to hit me up with any questions or feedback.

Great point! It was originally about both of these things, but the storage aspect isn't discussed much anymore because it's really not a concern.

The data integrity issue is still a concern, and I talk about that in the book. You need to manage data integrity in your application and think about how to handle updates properly. But it's completely doable and many people have.

I think there are some features of DynamoDB that are miles ahead of other databases:

- Billing model. Pay for reads and writes directly rather than trying to guess how your queries turn into CPU & RAM. Also able to scale reads and writes up and down independently, or use pay-per-use pricing to avoid capacity planning.

- Permissions model: Integrates tightly with AWS IAM so works well with AWS compute (EC2, ECS/EKS, Lambda) with IAM roles. Don't need to think about credential management and rotation.

- Queries will perform the same as you scale. It's going to work the exact same in testing and staging as it is in prod. You don't need to rewrite when you get four times as many users.

A lot of folks are worried about migrations, but they're not as bad as you think. I've got a whole chapter on how to handle migrations. Plus, one of the examples imagines that we're re-visiting a previous example a year later and want to add new objects and change some access patterns. I show how it all works, and they're really not that scary.

Thanks! Honestly, I'm right with you. I would love a physical copy. I did a bit of research and didn't find any great options for making a physical copy of a self-published book for the number of copies I'm expecting to sell (given it's a fairly niche technical area).

That said, if anyone has any great recommendations here, I'm all ears. Actual experience would be best if possible, rather than the first thing you see in Google :).

The concepts are definitely applicable. You'll need to do some small work to translate vocabulary and work around a slightly different feature set, but most of it should work for you.

And yep, Cassandra is pretty similar to DynamoDB. Both are wide-column data stores. Some of the original folks that worked on Dynamo (not DynamoDB) at Amazon.com went to Facebook and worked on Cassandra. The concepts underlying Dynamo because the basis for DynamoDB at AWS.

For more background on Dynamo, check out The Dynamo Paper: https://www.allthingsdistributed.com/files/amazon-dynamo-sos...

If you want a look at the Table of Contents, check it out here: https://s3.amazonaws.com/dynamodb-book/table-of-contents.pdf

For a third-party breakdown of the book structure and content, check out Shawn Wang's review here: https://www.swyx.io/writing/dynamodb-book/

Shawn lists a four-part breakdown that's pretty on-point:

- Background and basics (Chapters 1-6)

- General advice for modeling & implementation (Chapters 7-9)

- DynamoDB strategies, such as how to handle one-to-many relationships, many-to-many relationships, complex filtering, migrations, etc. (Chapters 10-16).

- Five full walkthrough examples, including some pretty complex ones. One of them implements most of the GitHub metadata backend. Nothing related to the git contents specifically but everything around Repos, Issues, PRs, Stars, Forks, Users, Orgs, etc.

The first nine chapters you can probably find available if you google around enough. It's helpful to have all in one place. But the last 13 chapters are unlike anything else available, if I do say so myself. Super in-depth stuff.

I think it will be helpful even if you've been using it for a while, but it really depends on how deep you went on DynamoDB.

Thanks! Yea, Rick is awesome and really helpful. He wrote the foreword to the book as well :)

Rick has had the most-watched re:Invent talk the last three years.

Here are the links for each:

- 2019: https://www.youtube.com/watch?v=6yqfmXiZTlM

- 2018: https://www.youtube.com/watch?v=HaEPXoXVf2k

- 2017: https://www.youtube.com/watch?v=jzeKPKpucS0

Additional DynamoDB links here: https://github.com/alexdebrie/awesome-dynamodb

Author here! If you want more, I just released a book on DynamoDB yesterday --> https://www.dynamodbbook.com/ . There's a launch discount for the next few days.

The book is highly recommended by folks at AWS, including Rick Houlihan, the leader of the NoSQL Blackbelt Team at AWS[0].

Happy to answer any questions you have! Also available on Twitter and via email (I'm easily findable).

[0] - https://twitter.com/houlihan_rick/status/1247522640278859777

Writing.

It helps to clarify your thinking. At your job, it will help you explain concepts to others and multiply your efforts.

Externally, it can help make great friends and boost your career if you blog & share things you've learned.

Original author here.

The two sentences right before what you quoted are helpful:

"I’m not a Docker or Flask performance expert, and that’s not the goal of this exercise. To remedy this, I decided to bump the specs on my deployments.

The general goal for this bakeoff is to get a best-case outcome for each of these architectures, rather than an apples-to-apples comparison of cost vs performance."

I wasn't trying to squeeze out every ounce of performance and determine the minimum number of instances to handle 100 req/sec. I was trying to normalize across the three patterns as much as possible to see best-case performance. I didn't want resource constraints to be an excuse.

I think you're right that Fargate and Lambda often serve different use cases.

However, I think containers and Lambda can and do serve this particular use case -- handle an API request and forward it to a different system. And Fargate is a good stand-in for containers generally. I wouldn't expect much different performance by using ECS or EKS or EC2 -- it's still going to be a load balancer forwarding to a container instance.

Definitely not perfect, but I think it works as a general approximation. For this particular common use case, you have three options. Was curious to see the perf differences between them.

* Original author here.

Author here. I'd say this was about in line with what I'd see from using API Gateway and Lambda.

I thought about doing a price comparison, but that gets really tricky. At some point, you're testing the skills of the tester more than the services themselves.

Agree with your expectations though. I think Fargate is naturally faster and can be moreso given the number of knobs you can tune. This will likely cost you more, both in direct resource costs and in engineering time trying to fine-tune the knobs. Whether that's worth it depends on your business needs.

Author here. Yep, that was my suspicion as well so I didn't hold it against Fargate. I was more looking for a reasonable expectation of performance, so I just threw a ton of resources at it to rule out as much as possible.

One of the AWS container advocates mentioned it was likely something like the nofile ulimit: https://twitter.com/nathankpeck/status/1098992994131283968

You can use Kinesis Firehose to stream data to S3. It'll buffer data for a while -- you set thresholds based on size of data or time, whichever is hit first -- then it will save the data to S3.

It won't be a single large file, but they'll all have the same prefix based on date. Most data processing tools will let you suck up an entire prefix and treat it like a single file.

To me, it's easiest to think of it as "Make, for data". Your ETL pipelines are often a complex graph of dependencies. If some step halfway down the chain fails, you don't want to restart at the very beginning -- you want to restart where it failed and keep moving.

The ETL frameworks (Airflow, Luigi, now Mara) help with this, allowing you to build dependency graphs in code, determine which dependencies are already satisfied, and process those which are not. They'll usually contain helper code for common ETL tasks, such as interacting with a database, writing to/reading from S3, or running shell scripts.

Noted and updated! Thanks for the tip.

* Disclosure: I work for Serverless, Inc.

> it was relatively easy for me to write a really simple Express server, transform the Express requests into the API Gateway format, and convert the API Gateway response format back into Express format.

For those interested in this route, there's a plugin[0] for the Serverless Framework that handles this conversion for you. Honestly takes 1 minute to convert an existing Express app to serverless.

Shameless plug but I wrote a blog post walkthrough for anyone interested --> https://serverless.com/blog/serverless-express-rest-api/

[0] - https://github.com/dougmoscrop/serverless-http

Can you give more background on what your data pipelines are like? Are they mostly batch processes?

If so, I'd strongly recommend using a workflow tool like Luigi[0] or Airflow[1]. In a phrase, I'd say they're like "Make for data".

[0]: https://github.com/spotify/luigi [1]: https://github.com/apache/incubator-airflow

Clinkle: https://www.forbes.com/sites/ryanmac/2016/01/22/clinkle-up-i...

SEEKING WORK | Nebraska | Remote

Unlock your data -- Data Science for Hire.

We are a three-person team looking to help startups get the most of our their data. Our Harvard-trained data scientist can assist you at any stage in your data analysis project, from designing your algorithms to setting up your data infrastructure.

We can tailor a solution to what you need, whether it’s one-off processing of legacy data, centralization and standardization of your current data flow, or display of your data in a web app format.

We have experience with Hadoop and related tools, such as Spark, Pig, and Hive. We’ve worked with Postgres, Mongo, and Redis, and can quickly pick up any comparable tools you use. Our general language of choice is Python, and any web app would use Flask or Django, depending on the particular project needs. We have experience using D3.js for interactive visualizations. Finally, we’ve deployed on AWS and Digital Ocean and have toyed around with Ansible and Docker (with Fig).

Interested? Contact us at dontcoastdata [at] gmail [dot] com.