Shell Has a Forth-Like Quality (2017)
oilshell.org
oilshell.org
Oil has made a lot of progress in the last 3 years, with a few releases every month.
http://www.oilshell.org/releases.html
The goal of the project is to replace bash!
If you know shell and Python (or C++), I maintain a "help wanted" bug label:
https://github.com/oilshell/oil/issues?q=is%3Aissue+is%3Aope...
It's almost there, except for speed, but it needs more testing from users. The "should-run-this" label has some ideas:
https://github.com/oilshell/oil/issues?q=is%3Aissue+is%3Aope...
There a lot of project updates on https://oilshell.zulipchat.com/ too (requires login)
I am hesitant to spend the time ramping up on something like this because, as much as possible, I try to write my shell scripts so that they can run without any additional packages in an Alpine docker container.
Is this resistance that you have encountered before?
I've definitely seen people who want to limit themselves to either bash or POSIX shell on purpose, and that's fine.
For simple shell scripts, I might lean that way too (at least until Oil is smaller/faster).
One major reason for Oil is to give you an upgrade path after you've used and enhanced such a shell script for years. Then you will likely bump up into shell's limitations, and the extra dependency will be worth it.
For example, all Linux distros are based on thousands of lines of shell scripts [2], and I think those scripts desperately need a new language. They're big and complex, and need a better language to be correct, robust, and fast.
Here's a comment I wrote that might explain the design goals of the project more:
https://www.reddit.com/r/ProgrammingLanguages/comments/ddhmj...
[1] Oil Language Design Notes #1 http://www.oilshell.org/blog/2019/08/22.html
[2] Success with Aboriginal, Alpine, and Debian Linux http://www.oilshell.org/blog/2018/01/15.html
I agree a lot of this stuff hasn't been documented well in existing shell resources. I should write more about it, but I'm still busy working on the Oil shell implementation nearly 3 years later :) I should have written a post about xargs 2 years ago, which has nontrivial compositional patterns. Hopefully next year!
I still recommend The Art of Unix Programming (mentioned in this post), which is more generally about Unix design, not about shell in particular.
I also got a lot of ideas by reading DJB's shell and C code, and his papers (e.g. Ten Years of qmail Security). I probably learned about DJB 10 years ago from Hacker News.
I would say there's a lot of "tribal knowledge" in people's heads, and online forums are where it gets disseminated. For me those forums were Usenet, the old JOS forum, and Hacker News and lobste.rs these days.
----
Neil Stephenson has a good quote about this "oral history"
Unix, by contrast, is not so much a product as it is a painstakingly compiled oral history of the hacker subculture. It is our Gilgamesh epic.
https://en.wikipedia.org/wiki/In_the_Beginning..._Was_the_Co...
https://en.wikipedia.org/wiki/Epic_of_Gilgamesh
The next post has the "point-free" pipline pattern which is both useful and hard to find out about!
hist() { sort | uniq -c | sort -n -r; }
Pipelines Support Vectorized, Point-Free, and Imperative StyleAnything complex, will require looking up some arcane syntax, which the next person or you in 2 years, won't understand what actually does. So will you notice subtle bugs or be able to upgrade or fix the code? The result will be bitrot or adding complexity.
The nice things with shell scripts is how fast you can pipe something through UNIX, and it's really really powerful. But regard it as temporary or throw-away code, unless it's just meant to sit there doing the same job, with the same subtle bugs that you won't notice if it's anything more complex.
But I prefer the shell code for tasks interfacing with system tools. I wonder if there is a shell interpreter around that is really fast.
https://github.com/oilshell/oil
Oil isn't fast at the moment, but I'm currently optimizing it (auto-translating some code to C++, and eventually hand-optimizing other parts.) I run benchmarks on every release, so speed is definitely a goal.
I haven't seen that many shell scripts that are "CPU bound" (as opposed to waiting for processes or I/O). (Although now I recall a shell-script-library bundler project that was.)
Of course, one big problem with shell is that you often have to spawn an external process like sed or awk to do string manipulation that's just a library call in Python. That could easily be a three order-of-magnitude difference -- e.g. milliseconds vs. microseconds.
I'm still interested in more feedback on Oil (e.g. does it still work on the latest version of shell-script-library? :) )
It's changed a lot, and the Oil language now exists! Latest post: You Can Now Try the Oil Language -- http://www.oilshell.org/blog/2019/10/04.html
Next time I’m working on it I’ll grab a release of oil and see how it fares!
I remember seeing something like what you are doing a while back, but it was such a small project that I can't recall its name or find it via Google. But I think it was a shell favoring functional programming style meant for plan9.