Did you ever want to find an alternative to cloud data PaaS/SaaS vendor lock-in?
In my blog post, I show how a 0-cost switch between local, Databricks and AWS EMR environments can be achieved to save cost by using dagster. Additionally, software engineering best practices around testability, and producing DRY code are enhanced.
Check out my deep dive and let’s discuss the potential impacts on our projects!
Big thanks also to my co-author Hernan.
The idea of running workloads on the cheapest compute platform is interesting.