I have started using prefect.io as a workflow runner. It helps me define tasks and deeply define what should happen on failure of a task including cleaning up a cluster.

Because Tasks are just functions you could test your tasks with regular unit testing and mocking methods.

But I agree a lot of data people seem to think interactive is good enough.