Definitely agree that that is one of the great points with Luigi. Airflow's UI of course blows everything out of the water IMO.

As for organizing your data, my personal and very biased opinion is that version control semantics similar to Git [0] are a pretty good way to help tame the complexity of ever-changing data sets. We already version code, but with versioned data too, now everything becomes completely reproducible.

[0] Our Data Science Bill of Rights: http://www.pachyderm.io/dsbor.html