Oil Shell: Success with Aboriginal, Alpine, and Debian Linux
oilshell.org
oilshell.org
A shell script has a very simple function: execute a number of tedious but common commands. That's it. Unfortunately someone decided they wanted control flow, so they wrote the horrifying [ and [[ programs, which I still consider to be a crime against humanity. Enter decades of write-once-read-never scripts that run with abysmal efficiency, kill maintainability, and neuter innovation. And somehow the entire Linux ecosystem depends on it and now every distro has to ship a shell with cruft dating back to 1970, with probably as many hidden vulnerabilities. And now every shell that tries to position itself on the market has to implement the same quirks, bugs, and design flaws in order to even be considered.
Let shells be shells, let shell scripts be dumb as bricks, and please use python or even perl for the love of god if you're going to make a build system.
Please read this comment: https://www.reddit.com/r/linux/comments/7lsajn/oil_shell_03_...
It's not a small amount of shell, and it's not even old in some cases. People are still writing big pieces of shell code. I point out at the end of this comment that Kubernetes, a brand new cluster manager from Google (2014 or so), has 48,000+ lines of shell in its repo.
Shell is definitely a terrible language for many things, but that's what I'm trying to fix, in a way that doesn't force you to rewrite all your code. Some things will never be rewritten -- it's like saying "Hey why don't you rewrite Wikipedia in Python and get rid of all the PHP?" It will never happen, for fundamental reasons of economics.
Also see this comment for more of the "why":
https://www.reddit.com/r/commandline/comments/7c3f9f/osh_02_...
I plan to write these up on the blog, as mentioned at the end of this post.
Consider this response for entertainment purposes only. It’s easy to become insult(ed|ing) or get into a holy war, but I’m really not invested in any of that.
That said: it is a useful sentiment, and a kubernetes script written in sh(1) is not the answer to the question.
That one “can” does not mean one “should”.
The reddit comment cites init scripts (I happen to use RC scripts, and haven’t viewed init scripts in a while) as an example of why shell is so great - everybody uses init!
1) of course everybody does. And it -happens- to be written in shell. So what?
2) this is fine! The scripts I see are 10s of lines long and practically flow top->bottom w maybe one or two conditionals.
This is more akin to a job control language. A perfect application for sh(1).
I just feel it (and you can too): sh(1) (or bash - nobody cares) features feel archaic and brittle. People have written impressive work in brainfuck[0][1], or C that just emits mov[2] instructions; is that a case for brainfuck projects or mov-only assembly? No. It’s just interesting. And kubernetes scaffolding or anything else big that happens (by some miracle) to work are not excuses for huge shell scripts. That’s just tautology - otherwise we’d be talking about the virtue of these scripts. For a boot sequence (rc/init) even if we ignore that the short scripts are actually suitable for the important job, I’d say their other grace is that sh(1) is demanded by POSIX, so you don’t need to wonder if it’s available in this nascent state of booting your OS - you’re guaranteed. The same couldn’t be said for ruby or python.
From the “see this comment” comment:
> Have you ever written a bash completion script yourself? I did a few years ago and I am scared to ever touch it again! I have a non-deterministic bug that I can't figure out, so I just restart my terminal every time completion gets borked. And I know bash very well.
The comment admits what a warzone the space is... Sure, fix it, but we don’t need to promote it.
The sentiment of discouraging huge shell scripts the GP expressed is useful. Resist.
[0] https://en.wikipedia.org/wiki/Brainfuck
Maybe some shell features are not that great, but in general I feel the opposite. Many of the shell features are great and lacking in other, more general-purpose languages - for example, program composition via pipes, stdout/stderr redirection syntax, direct access to the filesystem, interchangeability of a function and a program call, and others.
When somebody decides to write a big system, like Kubernetes, they better be pragmatic rather than idealistic. That means choosing the right tool for the job, even if this tool has some deficiences or is considered quirky by some people. Manipulating VMs,containers on localhost or remote hosts, interacting with the OS and the filesystem, all on a unix-like system, it seems to me like shell is the best tool there is for most of the task.
That’s a good, big one in favour of shells.
Re: file system, std err/out, I’m not convinced, but pipes alone are worth a lot. I wasn’t thinking at all of pipes, and you’ve tipped the scales, but is it enough? Do Tcl/Perl/ruby/python/... compete in that space?
How does the existence or lack of a standard data interchange format affect the value of those pipes?
We pipe everyday, all day, and the world carries on, so that’s a pragmatic vote of confidence for pipes.
Are you trying to get me to write a “worse is better” essay that ends with “...and the existence and continued durability of shell scripts is proof enough that their place is not only justified, it’s vital!”, because it won’t work. Pipes are good. Shell scripts should be looked at suspiciously if they’re more than 20 lines long.
;)
I do wish there was a first class syntax for it in Python (but you can get most of the way there by using generator/coroutine pipelines).
Pipelines Support Vectorized, Point-Free, and Imperative Style
http://www.oilshell.org/blog/2017/01/15.html
I started writing a series on "what shell can do that other languages can't", but I put it on hold until the shell is in better shape. I hope to resume that series in the next few months.
(1) In retrospect, I should have responded to the first commenter differently. He was clearly angered by having to debug other people's shell scripts in the past. But that is exactly what I'm trying to fix -- I'm trying to provide an UPGRADE PATH OUT OF BASH. I explicitly state that in this long post, which I think some people have missed.
These posts show one part of the plan: http://www.oilshell.org/blog/tags.html?tag=osh-to-oil#osh-to...
(2) I'll repeat my objection: trying to convince people not to use bash is going to be about as successful as convincing people not to use PHP. Not only is it not a solution to the problem, it also ignores the humongous installed base of PHP code, like Wikipedia, etc.
Even if another line of bash never gets written, you'll still have to deal with it on a regular basis!
I am providing a path out of bash, while other people are just wishing the problem would go away. The world is not how you want it to be, but exhortation on online forums isn't going to change the world. Less charitably, "you're wasting your breath".
I need to write a blog post entitled "Reimplementing Bash is the Only Way to Kill It". This is analogous to how Facebook is "removing" PHP from their codebase by developing the Hack language and VM.
It's also analogous to how Microsoft "killed" Lotus and other competitors by understanding their file formats.
(3) I also don't think you understand why cloud projects like Kubernetes are using shell scripts, probably because you don't work in that domain. Shell really is the best tool for that job.
I think it's presumptuous to state that the people writing that code don't know what they're doing. A lot of the project is in Go, but a lot of it is in shell, and there's probably a reason for that other than ignorance.
Re: 3) above -- Curious to hear about the reasons.
Re: "...people writing that code ...other than ignorance" -- I'm not trying to imply people working on this are idiots... though w/ broad-sounding statements like "scripts with more than 20 lines should be looked at suspiciously" I can see how someone might read that. I think there are myriad poor reasons an shell script might be inappropriately used. There's also a case for "if it's not mine, who am I to judge?". I'd be curious to hear points of why shell is especially suited for (e.g.) cloud projects like Kubernetes, though.
Good luck with your work.
I think you need to actually do it to have an appreciation for this. Other people here have made the same points I'm making, but if you don't try it, you won't viscerally understand it. I would just caution against the attitude of assuming that people who use a tool you don't like / don't want to learn (understandably) are using the wrong tool for the job.
As mentioned I should write a blog post called "Python Is Not an Acceptable Shell", in part based on this post, which is very in detail, but I think proves the opposite point:
https://medium.com/capital-one-developers/bashing-the-bash-r...
Of course, bash has significant downsides for this task too. That is why Oil exists!
Please explain why. Because simple observation seems to prove the opposite.
Zsh, especially with 'set -eu', is a perfectly suitable, albeit slow, replacement for Perl, which is in my opinion more distasteful.
It helps that the Arch developers are good at writing clean shell scripts.
[1] In the package "arch-install-scripts".
I prefer pure hand-written shell to an unholy mix of auto-generated shell and Make. The latter is what I remember Debian's build system to be, although admittedly I haven't looked at it in awhile.
I wonder what the reason was.
I'm not sure if apk used to be in shell, but I wouldn't be surprised if it were. If you have a link to the old version I'm interested.
I think the main reason not to use shell for the user side is the version solver. To install packages, you have to solve dependency constraints, which is actually an NP-complete problem! Those heuristics are best coded in C.
In contrast, the build side doesn't have to do anything like that, or it can just shell out to "apk" if it does.
> A shell script has a very simple function: execute a number of tedious but common commands. That's it.
And why is that? What will happen to me if I write more complicated shell scripts? The functionality is there and one can write very expressive and readable programs in shell. I would definitely recommend shell for writing long programs (for example various system administration and web backend tasks which do not require complex libraries). Preferably bash since it is the most common one.
> And somehow the entire Linux ecosystem depends on it and now every distro has to ship a shell with cruft dating back to 1970
The 'somehow' has quite a simple rationale, it isn't something that should outrage you. The family of Linux-based OSes take inspiration and compatibility target in unix systems on purpose. Unix systems are primarily operated via shell.
The current standard of shell may be an old idea from 70's with some 'cruft', but what language does not have cruft? The shell has proven to be usable and worthy of continued use. The compatibility with the unix shell is a big part of the early success of GNU/Linux and majority of users need it.
> And now every shell that tries to position itself on the market has to implement the same quirks, bugs, and design flaws in order to even be considered.
I have no idea what you are talking about.
> please use python or even perl for the love of god if you're going to make a build system
OK, so you think perl and python are better for making a build system. Have you made one?
As part of my job currently, I am going to delete it.
The same goes for poor python code, but I will take bad python over shell scripts any day of the week.
Go away, or I will turn you into a very small shell script.
#!/bin/bash
find / -name \*.sh -size +10k -type f -print0 | xargs -0 rm $ mangle find -delete
ACTIONS
-delete
Delete files; true if removal succeeded. If the removal failed, an error message is issued. If -delete fails, find's exit status will be nonzero (when it eventually exits). Use of -delete automatically turns on the
-depth option.
Warnings: Don't forget that the find command line is evaluated as an expression, so putting -delete first will make find try to delete everything below the starting points you specified. When testing a find command line
that you later intend to use with -delete, you should explicitly specify -depth in order to avoid later surprises. Because -delete implies -depth, you cannot usefully use -prune and -delete together.Im pretty sure you meant to say "POSIX, because its rules are supported by multiple shells, and you can't depend on bash being available or a recent version on some systems".
For commercial unices or bsd's bash could be hard to use and using POSIX shell may make more sense.
I'm definitely pro on shell scripts, but I'm also willing to use a different language when posix SH isn't going to cut it.
Not all have bash. Not all have python. More probably have perl than python, so if you want something other than sh, then perl should probably be it.
Sure it's not as great as a full on CI server, but it doesn't require the same investment either. For my personal projects I have bash scripts to handle basic CI (compiling and testing), given a bit of time I think I could string cron, make, bash and m4 into a CI server that fits 90%+ of my needs.
http://www.oilshell.org/blog/2017/01/13.html
(I didn't come down on either side of the debate -- it's more of an exploration.)
I'm just tired of bash getting bashed everytime someone mentions shell scripting, based on what mostly seems to be bandwagon reasoning. Of course if you are writing more than $arbitrarycomplexity drop into perl, etc, but there is so much that can be done quickly, easily, and readably even for a non-programmer like myself that I think there are many benefits to using shell scripts that get far too often ignored.
Also, a perfect example of a great bash script that's over 2k loc, and I've got plenty more were that came from: https://github.com/centminmod/centminmod/blob/master/centmin...
Now I'm not as pushy as OP, but it's far from baseless bandwagon.
I challenge you to reimplement the shell script that grandparent linked in 50 LOC. In fact, shell script has the chance to be a lot more succinct than other languages for a lot of administrative tasks since the invocation of other programs is a first-class citizen. For example, compare this Go snippet:
cmd := exec.Command("apt", append([]string{"install"}, dependencies...))
cmd.Stdin = os.Stdin
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
err := cmd.Run()
if err != nil {
fmt.Fprintln(os.Stderr, err.Error())
os.Exit(1)
}
if ! apt install ${DEPENDENCIES[@]}; then
echo "exit status: $!" >&2
exit 1
fi
Finally, what is it with sysadmins maintaining large scripts while at the same time quipping that they are not programmers? A script is very much a program, and a scripting language is very much a programming language.
You are correct, but I mostly use this preface to let people know I don't have the detailed formal training in programming most devs do. So while I can wiggle my way around perl/python/bash/awk/sed and other "systems languages" just fine I am constantly finding out how much I don't know.
I suppose in the early hacker days sense of the word most of us are programmers.
You think that once you have a CS degree, it's not like that anymore? If anything, that feeling gets more and more common over the years.
It's true that calling programs shortens tremendously, but every time you have to glue things together you're back to grep/sed and all the fragile bash idioms.
Of course you could have standalone programs to do that, in the end it would be as having a non bash programming language in disguise
We've met some programmers, and don't want to be compared to that[1].
Seriously.
Programmers will embark on some massive mission that Apache and dumb CGIs can do[2]
Programmers will justify their slow software by saying it's readable, or by saying that the JIRA ticket didn't specify how fast the solution needed to be.
Programmers will make user interfaces that require clicking multiple times to find some bug, then require the report tediously repeat those clicks, complain that the bug report wasn't accurate (can't find a button that says "OK" when it says "Close", and there's no other buttons).
Programmers will actually shit in the pool, and say it improves readability. Seriously: I have seen programmers delete perfectly working code and replace it with something broken (especially on the edge cases) and then ask the sysadmin team to wake up at 3am to restart their software every night. Code that was working fine for years. Code that wasn't even in their project. Fuck that.
[1]: https://news.ycombinator.com/item?id=16155641
[2]: http://marcio.io/2015/07/handling-1-million-requests-per-min...
I do lots of other things l, like make sure my software works, and make sure the users are successful with it — but I don’t know what that job title is.
“I print money for my employer” seems a little OTT...
The point is that I've been programming for over three decades, and over time this skill became just one of many things that I do.
"Programmers", are people where that's all they do, so they look to solve problems with more programming. Wanting to call them "software engineer" or "software developer" or "potato" anything else doesn't matter to me.
rewriting for the sake of it, with regression is plain bad
also just in case, I'm not saying sysadmin are lower; but their job is not to learn about programming languages and paradigm, so they don't know what goes on there. Now, that they can make good programs, sure. Having sound logic is not limited to "programmers".
Sure, so the fair comparison to make is with calling library functions - all those other programs had to be implemented separately too. If I were reimplementing the linked script I'd use something like Puppet where "ensure package installed" is a first-class citizen too.
That's patently false: https://fishshell.com
https://github.com/fish-shell/fish-shell/wiki/POSIX-compatib...
#!/bin/bash
trap '(( LINENO < 10+3 )) || { echo "Script executed more than 10 lines of code! Aborting."; exit 88; }' DEBUG
echo 1
echo 2
echo 3
echo 4
echo 5
echo 6
echo 7
echo 8
echo 9
echo 10
echo 11
$ ./test.sh
1
2
3
4
5
6
7
8
9
10
Script executed more than 10 lines of code! Aborting.1. Re-implement Bash and similar shells in a modern programming environment with an eye to extensibility
2. Make good debugging tools because this is now easier because (1).
3. Create a new language that extends Bash, similar to how ES6 extends ES5, for a gentle migration of large shell scripts to a safer language
What, other than static parsing, should this endgame language offer? Where does it fall, on the gamut from Bash to Haskell? We already have quite a few languages that offer a souped-up scripting experience.
When my bash programs become too long, I usually reach for Ruby, which has convenient back-ticks syntax for shelling out, and many convenience methods on its built-in types for mashing strings and such. But like Bash, too much ruby eventually becomes an unmaintainable mess without strict standards enforcement. What will Oil offer for me, over Ruby?
Here's how I'd order existing languages on a scale from Bash to Haskell:
1. Bash with `set -e`
2. Perl
3. TCL (has declared arguments, after all...)
4. Ruby
5. Python
6. Golang
7. Java
8. C++
9. Ocaml
10. Haskell
Then you could start refactoring code to it. Easier than a rewrite.
I got quite far along at building a message-driven paradigm in awk. The input is a stream of well-defined messages. As you read it, append new messages to the end of the stream. I found that the awk I was using (gnu I think) was complicated by IO buffering behaviour. Still, an interesting problem space.
Lex may offer strong opportunities too, although is not part of the posix base.
I somewhat agree with your spectrum of strictness, but the other dimension is "how suited is this language for shell-like problems?".
C++ is strict, but obviously not good for writing "shell scripts" in.
Oil will be somewhere around Python in terms of strictness, but at least as suitable as bash for shell-like problems (of course). The goal is for it to be more suitable, but I haven't gotten there yet.
As far as what it offers, that is scattered over the blog, but this tag might give an idea:
http://www.oilshell.org/blog/tags.html?tag=oil-language#oil-...
As a concrete example, Oil will be based more around arrays of strings than flat strings. The "Thirteen Awkward Ways..." post hints at that. FWIW, the Ion shell, part of Redox OS, was influenced by this way of thinking.
Also, I need to write a blog post entitled "Python Is Not an Acceptable Shell". There was a nice blog post that translated shell to Python that inadvertently proved this point.
I'll keep your questions in mind when writing future posts explaining the project.
I had a shell script on my desktop that would automagically extract any archive I threw at it. Nice. So when I was working on android through a shell, I wanted the same convenience and so I scp'd the script onto my phone. It wouldn't run. Only worked with GNU tar, as opposed to any other tar, like bsd tar or busybox's.
Having a language that provides system-independent abstractions of ls, tar, top, or the other way, that clearly indicates upon which binaries it depends, would greatly improve script usability.
It'd be nice if running an oil script would automatically warn me that I need GNU tar >= version x.xx.
I don't think you can just compare languages that way. There's more than one axis. For instance, to say python, then java, then c++ seems backwards in terms of writing scripts because c++ is much more suited for that. But it's also backwards for the purpose of writing applications, because they're all around the same place, albeit in different ways. It's not so simple as saying "here are these languages, here's where they lie on a scale from bash to haskell", because there is no scale from bash to haskell.
I have been enjoying "Beginning Portable Shell Scripting"[0] by Peter Seebach for this very reason.
$ cat run-bash-script.sh
#!/bin/sh
which bash &>/dev/null && exec bash "$@"
source /etc/os-release
case "$ID" in
debian) apt install bash || exit 1 ;;
redhat|suse) yum install bash || exit 1 ;;
arch) pacman -S bash || exit 1 ;;
alpine) apk add bash || exit 1 ;;
esac
exec bash "$@"
Though in many of these cases I'd just advise leaving the premises unless the salary is stellar (or there are other considerations such as family, etc).
I highly recommend it (and the POSIX docs) to anyone who is is interested in portable shell programming.
---Alex
Since I have my hands full with the distros I'm using, I may not be able to try it for awhile, but I saved this page for further reading. I still think distros are somewhat "broken", so alternative approaches are interesting to me. (Just one problem off the top of my head: packages are old, distros can't keep up with PyPI, npm, CRAN, etc.)
If you know the SourceMage code, please try it with OSH :) Or if you know its developers, they might want to hear about a cleaner shell that's compatible with bash (although it's nascent)
I spent some time last fall implementing some constructs that the Nix package manager uses, even though I don't use Nix. Nix uses bash heavily as well.
That said, I think you've highlighted _the_ problem that _all_ distros face that would get me to change distros if someone solved it. That is to say, “(Just one problem off the top of my head: packages are old, distros can't keep up with PyPI, npm, CRAN, etc.)” That is a doozy of a problem.
One last thing–you mention four things:
The trap builtin is unimplemented;
alias is also unimplemented.
set -h / hashall is a stub that does nothing.
OSH builds are in a sense "shallow".
Implementing `trap', `alias', `set -h' seem like no-brainers prior to 0.4
But yes I do think distros are doing the best they can with the raw material they have to work with (upstream sources, autoconf, etc.) I have some ideas for a distro but I think I have my hands full with shell, and it is a big problem because you can't do it by yourself. You need an army to help you maintain packages to some standard!
-----
For the shallow problem, I'll probably implement something like set -o hijack-shebang. It's relatively easy for OSH to open scripts it runs and check for a shebang that looks like a shell, and then prepend "osh" to the argv array.
It should really only be used for debugging OSH, because it wouldn't be good practice to "lie" about your shebangs. That logic is fundamental and baked into the kernel!
"trap" has come up a lot, although I'm not sure if it will be in the 0.4 release. I'm making releases pretty often -- every 6 weeks or so. "alias" is interesting because it sits between the lexer and the parser, and no other shell feature is like that.
I didn't mention that because it's not a good idea yet to make OSH the default shell on any system :)
Sounds great! You put so much effort into your blog posts, it's impressive…
http://www.oilshell.org/blog/2017/04/08.html
There are some detail here in these posts:
http://www.oilshell.org/blog/tags.html?tag=opy#opy
FWIW, Oil was in Python 3 at one point! But I ported it back to Python 2. There were a number of reasons, but unicode handling is one of them.
See this comment:
https://www.reddit.com/r/ProgrammingLanguages/comments/7elxl...
Summary: The shell deals with strings from file systems, and file systems inherently have no encoding. So Python 3's model of unicode doesn't help with such programs. It just makes things more awkward.