A Guide to Continuous Data Quality Monitoring for AWS Athena

lariatdata.com

35 pointsstreamdreams2 comments

If you are going to use Athena don't forget to setup a limit to your queries. It's not enabled by default and remember you pay by amount of data scanned.

"Setting data usage control limits" - https://docs.aws.amazon.com/athena/latest/ug/workgroups-sett...

Also make sure your follow the best practices. Again...Lots of money to be saved or wasted...Depending on how well you follow them.

"Top 10 Performance Tuning Tips for Amazon Athena" - https://aws.amazon.com/blogs/big-data/top-10-performance-tun...

Good points re: Athena performance tuning. I've found that my go to has been to do some form of ingest date partitioning, and use parquet compressed with zstd (among some other things mentioned in your links). Although with the improvements made in Athena version 3, I'm excited to test it out with Delta Lake tables!

Reply on news.ycombinator.com