Hacker News Reader

Comment by theamk | Hacker News Reader

This feels great until your program reaches some complexity threshold and you start hitting core limitation of the shell execution model. Then you have to use high-numbered file descriptors, named temporary files, PIPESTATUS, etc... - and your neat programming model becomes ugly and full of hacks.

Sometimes shell is really the best even for complex stuff - for the example in initrd or during machine provisioning - but in most cases, having any sort of non-trivial flow control in shell is a bad idea.

View on HN

> any sort of non-trivial flow control in shell is a bad idea.

It is often a bad idea in other programming languages also. Yet, some complex control flow constructs are straigthforward in shell but very difficult in other languages. How do you

    ls *.jpg | grep 2019 | while read x ; do
            echo convert ... $x album/$x
    done | xargs -L 1 -P 32

in javascript or python, for example?

Wow, there are 7 ways this can easily fail, and the script might not even notice.

How do you write in a normal language? With roughly 3 times the amount of keystrokes, but in a much more robust way. And if something fails, you will actually get a Python exception.

If I understand correctly, the whole point of Oil is to make a shell in which this kind of code is perfectly correct and robust.

Yes exactly, see my sibling response. This is a perfect use case for the "xargs $0 dispatch" pattern I mentioned, and maybe I will use it in that blog post.

Your example does have some quoting problems which is understandable / common, but in Oil the obvious way should be the right way. That's not true in shell!

https://news.ycombinator.com/item?id=22161847

Thanks again for the encouragement!

The only difficult part here is xargs parallel invocation. If you do this often, you may want to find/make the library for it. But even with just default python install, it is pretty straightforward:

     files = [f for f in glob("*.jpg") if "2019" in f]
     commands = ["convert ... {} album/{}".format(f) for f in files]
     subprocess.run("xargs -L 1 -P 32".split(), input=os.fsencode("\n".join(commands)))

Note that this code, as well as original shell version, breaks if the filenames contains spaces or quotes; this is trivial to fix in python, but hard to fix in bash.

In sh, the correct version would be

    for f in ./*2019*.jpg ; do
        convert "$f" album/"$f"  # just don't use xargs
    done

Note the prepended ./ which is necessary if you want to protect against filenames that look like command line switches (a file named "-rf" can be pretty dangerous)

If you wanted to use xargs and still make it perfectly safe with regards to spaces and quoting, you could use a GNU extension:

    find . -maxdepth 1 -name '*2019*.jpg' -print0 | xargs -0 convertalbum.sh

    # convertalbum.sh
    convert "$1" album/"$1"

find prints the files terminated by ASCII NUL characters, xargs can understand this format if you give it the -0 option.

There are still problems around detecting errors in pipes, though.

> just don't use xargs

the whole point of my example was to show how easy parallelism is. You create a list of tasks, then you pipe the list to xargs or gnu parallel.

You can't write ASCII NUL characters from portable shell, that's why this task cannot be done correctly (for all possible filenames) from shell. You need to use tools like find(1)

So, no, it's not easy, at least not if you want to handle all cases without wreaking havoc.

Practically speaking, you can write NUL from shell because bash is everywhere, busybox ash emulates a lot of bash, etc.

However I won't claim it's pretty or easy for people to use, so the point of Oil is basically to clean up patterns like this, e.g. find -print0 | xargs -0 and more.

This post uses a related problem and solution as design motivation for Oil:

How to Quickly and Correctly Generate a Git Log in HTML

http://www.oilshell.org/blog/2017/09/19.html

> Note that this code, as well as original shell version, breaks if the filenames contains spaces or quotes; this is trivial to fix in python, but hard to fix in bash.

Your code sort of cheats, since it is calling xargs to do the parallelism. Also, it is missing the imports. The equivalent pure python version will be probably still more complicated (regardless of it working for general filenames or not).

> Your code sort of cheats, since it is calling xargs to do the parallelism.

This is the whole point though -- you can still invoke external binaries in python!

For example, if you need to sort 100GB file, direct Python approach will likely OOM... while /usr/bin/sort will work just fine. I have seen people use this as an excuse ("I cannot use python's sort(), so I will just rewrite everything in shell") -- but for a complex script, it is almost always better to use Python as much as possible.

Reply on news.ycombinator.com