Dgsh – Directed Graph Shell | Hacker News Reader

Dgsh – Directed Graph Shell

www2.dmst.aueb.gr

135 pointspcr91030311 comments

Python's mario is another terminal-oriented tool with concurrent task execution, but with async concurrency rather than multiprocess parallelism.

For example, to get a bunch of urls concurrently

$ mario async-map 'await asks.get ! x.text' < urls.txt

https://github.com/python-mario/mario

Was mario inspired by spotify's luigi? Coincidence?

https://github.com/spotify/luigi

A thread from 2017: https://news.ycombinator.com/item?id=13352659

Thanks for posting that. There is good additional discussion there.

Edit: SciPipe looked particularly interesting. https://github.com/scipipe/scipipe

Thanks for sharing - I was previously pondering a bit how a generic graph pipe shell syntax would look - I might try this out myself soon :)

1980, operating systems class, write a shell for UNIX. I thought I'd be clever and do more than simple pipes, so I created some syntax to enable the specification of a directed graph. It's pretty neat to see that someone has now done this industrial strength.

I think it's a pretty natural idea. According to Brian Kernighan's memoir, the original vision for pipes was more like a mesh than the omnidirectional flow we ended up with:

"Doug wanted to allow arbitrary connections in a sort of mesh of programs, but it was not obvious how to describe an unconstrained graph in a natural way, and there were semantic problems as well: data that flowed between programs would have to be queued properly, and queues could explode in an anarchic connection of programs. And Ken couldn't think of any real applications anyway."

My first thought when reading over the docs for dgsh is always something similar...what's the real application? What does it gain me other than some mathematical elegance that I don't lose in harder to read scripts? The examples all pretty much collapse to doing things in parallel and collecting them via cat, which seems a little basic for something as abstract as a DAG.

Interesting effort. On the other hand there is a bunch of other more popular and battle tested tools from Apache Airflow to a bioinformatics pipelines managers, such as Nextflow, Snakemake etc.

Looks pretty interesting (automatically deleting duplicate files). Bookmarking to see if I can utilize this!

Dgsh sounds like dogshit... Sorry

Weird, my first thought when reading it was a character from Morrowind...

Reply on news.ycombinator.com