all: data.loaded data2.loaded
data.csv: wget 'some url'
%.sql: %.csv sed/awk/custom-python-script $< > $@
%.loaded: %.sql psql < $< && touch $@
I once built a World of Warcraft addon downloader & manager (including patching) with Make. My friends thought I was insane. Personally, I wondered how else you'd do it :-)
(actually, looking back at it, I apparently built it on plan9's mk (https://9fans.github.io/plan9port/man/man1/mk.html), but it's close enough)
If part of the data changes, or new data arrives, you just run make again, and the whole pipeline re-does whatever is new. Very slick.
It has free concurrency (make -j N). I've used this effectively several times for medium-scale (~GBs) science data processing.
It works surprisingly well, but it is definitely not the right tool for the job. After a number of years of adding/removing/changing jobs, maintaining it is turning into a nightmare.