Having used a variety of modern ETL frameworks in the past years, I consider writing a hands-on book about what I have learned on the way.
If I may ask, what questions do you find most difficult to solve in the context of real-world ETL setups?
If I may ask, what questions do you find most difficult to solve in the context of real-world ETL setups?
- https://github.com/getpopper/popper
- https://github.com/lyft/flyte
- https://aws.amazon.com/step-functions/
- https://github.com/spotify/luigi
- https://github.com/dagster-io/dagster
Another question is, I strongly feel like the definition of pipelines should not be in code, but in the database. I keep coming back to that design pattern every time I start coding my own simple scheduling solution. Is there merit to this thought?
As for the pipeline definition. One goal is to have a notion of pipelines that is both comprehensive and declarative.
As for a database, what would you store there? Container image to run? Past execution data (e.g. output path, time, errors)?
The software world has many pipeline-y things, such as CI definitions and these definitions usually live in configuration files.
What is difficult at time is the tracking of done tasks. Is the output a file or a new row in some database or many files or many rows or anything else?