Unix Shell Programming: The Next 50 Years [pdf]
sigops.org
sigops.org
How do you find where the script itself is? Because if the script comes with files, we want to find those when the script gets called from somewhere outside the current dir. Stack overflow suggests this as a first approach:
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
Dealing with filenames? Hope you enjoy pain. Spaces, newlines and quotes are a horrible pain to deal with properly, and regularly cause trouble.
Dealing with correctly passing parameters? More of the same.
Dealing with long file lists? You run into the command line length limit.
Obtaining some incredibly common system parameter? Some insane combination of grep, awk and cut, probably.
Personally I find it amazing that people invented this thing, then spent decades tripping on the same quoting and filename handling issues and not finding that to be a good reason to fix it.
These days if it's more than 5 lines of code, or if arbitrary filenames are involved I go straight to Perl or Python, because the amount of completely preventable pain is drastically reduced.
And somehow people think mastering this careless design is something to be proud of.
Can anyone name any other popular piece of software that is that relaxed about irreversibly destroying your data?
You are right it has some warts, but if a file is really that absolutely critical I make a .bak copy of it before I start my work. The shell has its limitations but so do all tools and the power it gives me is unmatched.
As for other popular software there is plenty out there but other popular software out there destroys something far more valuable to me. My time.
This seems to miss the point completely, unless you back up your entire hard disk before running every command. The whole problem is it's easy to nuke things you weren't intending to touch, accidentally.
The user-hostility of the shell is not a required feature to achieve its power. All of the advantages you cite could be had without being a terrible footgun with human factors mired in 1970s computing culture.
I jest but I do agree that Bash et al have their warts. Seriously obscene warts that only make sense when you look at the language from the content of 40 year old systems. But these days there are other options if you want the power of a command line but without (most of) the warts of traditional shells.
Only because people refuse to read the specification. The POSIX Shell & Utilities volume is exceedingly clear and straight-forward. (I vehemently disagree w/ its characterization in that paper.) It helps if you use the frames-based HTML version to make it easier to navigate and explore the various utilities, and especially to see how the syntax specification is broken down--while that section is short, it's difficult to appreciate without explicitly seeing how the subsections are related.
I only code in POSIX shell because I really can't be bothered to remember all the minutiae of other shells, almost all of which provide strong POSIX compatibility, Bash especially. I've never lamented the lack of arrays; but I regularly wish that POSIX would adopt ksh-style coprocesses so dash would have them. The shell is really all about stitching together processes (including subshell contexts), and coprocesses fill a conspicuous gap in functionality.
Bash has a different implementation of coprocesses that doesn't seem to be used much, and is very limited. I believe there can only be one coprocess active at a time, which is very un-shell-like! I got an e-mail once from someone who was using them, and I felt it could be achieved by other means.
I have a good idea coprocesses in Oil which basically is a startup time optimization. The coprocess will basically take argv over a unix domain socket, and have other commands to change the process state. It solves "the problem of VMs and containers that start slowly".
https://www.oilshell.org/blog/2021/04/release-0.8.9.html#wha...
Unix domain sockets are already used for the headless shell in Oil (more coming on the blog about that)
https://www.oilshell.org/release/latest/doc/headless.html#im...
Basically, you start a coprocess using the "|&" operator:
somecommand |&
cat <&p
"p" always references the most recently created coprocess, so to juggle multiple coprocesses you have to dup "p" to explicit descriptors:
exec 4>&p
somecommand2 5<&p |&
coprocesses provide a simple construct for cases where you would otherwise have to manually create a named pipe using mkfifo. The need for this pattern comes up surprisingly often, IME, yet because it's so messy--now you need to worry about /tmp races, cleaning things up, etc--people rarely bother, myself included. From the perspective of a shell it's a relatively simple construct to implement. The semantic benefit/cost ratio is exceptional.
This basic functionality is complemented with extension options to the read and print (non-standard) builtins: -p reads from or writes to the latest coprocess, and -u the specified descriptor number. I haven't used coprocesses much because I can't rely on them, but I don't think these are strictly needed. Maybe they were required for proper buffering in early implementations (or even current ones), where redirection to and from builtins didn't utilize the same buffered I/O context? That seems more like an implementation detail.
ksh93, bash, and some other shells provide the process substitution construct, "<()", which can provide similar capabilities to coprocesses in many cases. But I'm pretty sure anything you could do with process substitution you could do with coprocesses, making the latter the more powerful construct. coprocesses seems like the critical missing primitive from the traditional Unix shell model, and would round it out. (Of course, the sky is the limit for evolving the Unix shell model in new directions, as Oil has shown.)
ksh88 had coprocesses so I'm curious why it wasn't included in POSIX Maybe because bash lacked an implementation? Or incompatible quirks between ksh88 and ksh93 related to stdout inheritance across coprocesses?
At least in zsh, you can prevent such mistakes by
setopt NO_CLOBBER
Also, from bitter experience I've forced myself to get in to the habit of never typing:
rm *
cd ..
rm foo/*
In Bash it’s
set -o noclobber
set -Chttps://pubs.opengroup.org/onlinepubs/9699919799/utilities/V...
I didn't know about SawStop before, so your comment sent me on a wiki-hunt.
"Gross negligence" seems like a huge understatement for all the rationalizing and lobbying manufacturers did to avoid having to use safety technology.
https://en.wikipedia.org/wiki/SawStop#Opposition_from_trade_...
I am not a carpenter and context switching is expensive when it comes to thought work (probably because it’s still just typing, lacking the mechanics of swapping a hammer for a screwdriver).
Anyway, the point is by now, 20+ years into an IT career, I should have at least ONE option that can handle these needs without flip flopping through an entire operating systems “tools”.
The desktop metaphor is not the only real world model computing needs to get rid of. I want an interface that can do the job, not a million “tools”.
SCRIPT_DIR="${0%/*}"
the same applies with "combination of grep, awk and cut". awk contains almost all the functionality of grep, sed, and cut, so any pipeline of awk plus one of the others is almost always unnecessarily complicated. it would be like complaining "in python, string handling is so hard. look, you have to do import re; for x in re.sub(...).match(5).split(' ')[3:-5]: lst.append(x)". of course if you make it overly complicated then it will be overly complicated.
there are also too many footguns. spaces and newlines are trivial to handle in 95% of scripts, as long as you quote every expansion. as long as you never ever write cmd $file and always write cmd "$file", that solves virtually all whitespace problems. quotation marks in filenames are never interpreted in a special manner in Bourne-like shell unless you use eval, which has the same pitfalls as any other interpreted language (see python above).
SCRIPT_DIR="$(dirname $0)"
Just because we wish something weren't so, doesn't make it as we wish it.
This question does not really make sense. There need not be any script, or there may be many at the same time. What exactly do you want? Imagine, for example, that you are piping the output of some commands into the shell.
Sure, but that is irrelevant.
When there IS a script, you should be able to get its full path much easier, that is the complaint.
Doesn't need to act differently in what it does.
But it does need to report it, because e.g. you might want to bundle data/assets/config relative to the script, and want to be able to run it from wharever and have it still be able to find it.
In the later case, does it mean that it is not possible to write reusable code based on that feature? In the former case, what should be done when sourcing is done recursively? Can this be used to defeat file location, for instance, by generating a script in /tmp and sourcing another file from it?
Secondly, it's super frequent that scripts I write or that others write need the folder they're executed in as an argument.
99% of scripts I've seen aren't meant to be piped to. They're just launched as simple, dumb, standalone "programs".
The funny thing is that there about a dozen ways to spell it, like $(dirname $(readlink -f $0)). Since the shell itself doesn't provide this functionality, people come up with a lot of workarounds.
In Oil it will simply by $_this_dir, e.g. "source $_this_dir/mylib.sh" for relative imports. (Whenever a variable is silently mutated by the interpreter, akin to $?, Oil prefixes it with _).
https://github.com/oilshell/oil/issues/587
I solicit feedback on every Oil release: http://www.oilshell.org/blog/tags.html?tag=oil-release#oil-r...
so let me know what you think of the fixes: https://github.com/oilshell/oil/wiki/Where-To-Send-Feedback
Dealing with filenames is fixed in Oil: http://www.oilshell.org/blog/2021/04/simple-word-eval.html and with QSN:
http://www.oilshell.org/blog/2020/10/osh-features.html#safe-...
Again, if you don't think this actually solves the problem, leave some feedback. Oil is the most bash-compatible shell by a mile, so you might need to use it someday :)
(The theory is that in MOST situations you don't get to choose your language, just like you didn't choose C, C++, or shell. It's all the inertia of compatibility.)
Oil has named parameters: http://www.oilshell.org/release/latest/doc/idioms.html#use-n...
I know about the long file list problem, but I don't think I've ever run into it, mainly because I use xargs, which batches up commands correctly. If there is some other situation where it comes up, I'm interested.
Insane combination of grep / awk / cut: Oil has eggex which compiles to ERE to help you write egrep and awk patterns: https://www.oilshell.org/release/0.8.11/doc/eggex.html
You get syntax errors when you write the pattern (at parse time), not when you run it.
Crucially, we don't "rewrite" grep and awk. We just make them easier to use (optionally). There is a smooth upgrade path and you can retain your knowledge while forgetting about some sharp edges.
> Personally I find it amazing that people invented this thing, then spent decades tripping on the same quoting and filename handling issues and not finding that to be a good reason to fix it.
Yes I quoted David Korn complaining about the quoting problem in the early 90's, which was closer to Unix's invention than we are to the early 90's.
https://www.oilshell.org/blog/2019/01/18.html#slogans-to-exp...
The reason we still have it is the inertia of compatibility, and the fact that nobody really owns shell. Even though people rightly complain about Google's web stewardship, HTML5 was a great improvement and cleanup. They paid people to fix HTML, and it worked to a large extent.
Some food for thought: A Generation Lost in the Bazaar https://queue.acm.org/detail.cfm?id=2349257
https://github.com/oilshell/oil/issues/941
(This probably won't happen for a long time though, unless someone really wants to contribute and own it!)
It's a job control language after all.
The pipes and redirects, and the instant access to the file system is also very powerful.
Sure it could/should be improved massively, but I'd say it's still the best tool for a significant amount of pretty common use cases, it doesn't just suck at everything.
wget http://server{1..5}.com/file.txt >> /tmp/data.txt & wait && wc -l /tmp/data.txt
echo "2 + 2" | bc -lTotally agreed. A shell script exceeding that's about 10 lines of code becomes so complex to read, interpret and improve that even though it may perform faster I would rather get the job done in a language that I can understand better when I come back to this 3 months later.
Although ... on that note I don't think I'd ever choose Perl ha ha.
Hence everything has been stuck with the archaic limitations and oversights of posix shell or gnu bash as linux distros took over and largely standardized on it.
There have been numerous alternative shells and some significantly improve on the situation, unfortunately they've been largely ignored. Default shell choice has proven very sticky.
I do agree that it's practically impossible to write robust and safe scripts in bash/posix shell. You have to jump through many hoops if arbitrary user input is being handled. Frankly nobody should be writing generalized tools using such shells today, unless they have a very good reason to self-impose the constraint.
99.99% of everything is internet connected and auto-updates. New distro versions come out every 2 years.
There's no real reason why bash couldn't have been evolved into a sane language with sane parsing and with modern constructs. By sane, I mean... read this:
https://www.oilshell.org/blog/
At least these articles:
https://www.oilshell.org/blog/2021/01/why-a-new-shell.html (especially this one)
https://www.oilshell.org/blog/2016/10/20.html
https://www.oilshell.org/blog/2016/10/28.html
https://www.oilshell.org/blog/2016/11/06.html
If freaking Javascript could "use strict;" and turn a page, bash could have definitely have done the same.
It's just that there's no real commercial interest and FSF doesn't have the manpower (or desire) to do this.
https://www.oilshell.org/blog/2020/07/blog-roadmap.html#the-...
It turns out that bash's "shopt" can be co-opted for this, and that's exactly what Oil does. For example, after 'shopt --set parse_paren', you can do this:
if (x > 0) { # parens are now Python-like expressions, not subshell!
echo "$x is positive"
}
Oil is written in a way to make breakage less likely: with principled parsing and a detailed code representation using algebraic data types. I'm surprised at how much language evolution this style has enabled. It makes me optimistic about the project's future!
Although we still tons of work to do with the C++ translation -- writing a shell is hard either way!
True, but that's not a good reason for such a used application. It has been around for 40 years, I think it literally has trillions of runtime hours. It's used on billions of systems, most likely.
It's exactly the place where you'd create a super comprehensive test suite that covers <<everything>> (see SQLite) and then re-architect it to make it robust.
I wonder if it's due to lack or money or due to something else.
> https://www.oilshell.org/blog/
I think the point is that, it did evolve, as your link shows—and there are lots of other sane alternative shells, too. The problem isn't that they don't exist; it's getting people to adopt them in sufficient numbers.
Just wanted to point out that Hacker News runs on FreeBSD[0] :) Some people like the BSDs, and would appreciate it if everybody stopped designing for the Linux monoculture (well, monoculture if you disregard Windows and macOS).
[0] https://docs.freebsd.org/en/books/handbook/introduction/
"What HN uses" doesn't say much about tech trends :-)
Edit: You probably want to use Netflix as a better example. I think it's used by their CDN layer.
No program can do this portably.
But it's possible for an interpreter to tell the script it's interpreting where it lives, i.e. argv[1]. For example, Python does this in a pretty straightforward way.
Um, false?
Many people in industry use and love using the shell.
Please read, "In the beginning was the command line."
The shell is by far one of the most powerful and expressive tools we have. Yes, there are many ways in which it could be improved, but JSON? That's not it.
>> B1: Too arbitrary. The shell’s virtue of limitless composition (G1) is also its vice: the shell can compose arbitrary commands written in arbitrary languages.
This is what I fundamentally love about the shell, and you can try to pry it from my cold, dead fingers
>> B3: Too obscure. The semantics of the shell and common commands are documented in 300pp of standardese [7].
With respect to Bash, I couldn't agree more about the 'standardese' gripe. Man pages would be so, so much more useful if they prioritized showing examples of common/useful ways to use the command. I would love a 'man --examples $COMMAND' feature. Does something like that exist?
Obviously, replace 'tar' by whatever command you see fit. Works on any machine online with curl available.
I usually edit the man page to add cookbook/bugfix sections. Most of them are cookbook-style recipes. Whenever I find out some obscure bug or gotcha out (spaces in file names passed through a pipe is one of my frequent issues), I go to the man page of cut/tr etc. and add it to the page.
Next time I man for that, I remember what I did right a couple of years ago. Another thing I copiously modify is the "SEE ALSO" section for common commands, whenever I install new stuff from github etc.
I confess that it is not a proper solution, but it works for me. Man pages have continued in the gloriously terse initial style of Ken Thompson. For mere mortals like I, I need a nicer explanation. My machine's manpages for some commands (find, grep being the most modified) tend to look like the old "HOWTO"s on linux.
For example Kotlin/Clojure/Scala all started out focusing on interop with the base Java/JVM technology. They provide a new layer of modern language features that is still compatible with all the existing code without forcing a rewrite. Over time these new languages get adopted, some older libraries get a modern equivalent and that helps these languages to branch out into other areas: e.g. Kotlin Native, multiplatform and ClojureScript.
A similar thing can be seen with Typescript. It started as a type layer on top of JavaScript, making it easy to interop with old JS code and over time lots of pure Typescript library equivalents have popped up. Now that the Typescript ecosystem is flourishing we see technology like Deno that slowly moves Typescript away from its JS base.
For a post-POSIX shell to be adopted widely I think it will need to use a similar strategy.
Edit: re-reading my own comment, I might have just advocated for an Embrace, Extend, Extinguish strategy.
The only platforms where new languages kind of took over, the move was forced by the company that owns the platform.
However you are right, in every case even the system language for the platform got better.
For my part, I think the old syntax sucks and has got to go, and if your new shell's language is good enough, people will want to deploy it where they might use it, just like any other scripting language.
Even though it evolved from bash/OSH, and is in fact the same interpreter parameterized by some global options, it is a nice clean new language :)
It's surprising how much language evolution was enabled by "shopt" options, principled parsing, and algebraic data types!
I think the approach Oil takes is pretty cool. It just seems like a lot of work, and I'm much more interested in Oil than in OSH.
I can see how it might make Oil attractive to Linux distros for choosing as a system shell, though, since it means they could just ship the one binary, and then offer experienced POSIX shell users OSH as an escape hatch while still presenting Oil as the preferred way to write scripts.
It does also partially spare users from the problems Fish users face where CLI tools ship with scripts to initialize a bunch of env vars, and now suddenly you either need them to ship a Fish script, translate it yourself, or do some hacking where you run a BASH shell and mirror its env var changes.
So I can see some benefits for sure, but to me those are less exciting than the new stuff. :D
In the introduction, the paper mostly talks about the ergonomics / maintainability (that resonates the audiencel like me), but the proposed solution, "jash", addresses the performance problem (who cares???).
That's said, the brief survey in the first half of the paper is moderately interesting. It's not a real survey but an advertisement of the authors' work. But they've done something worth a quick glance like formalizing shell's semantics [1], which looks interesting if no useful.
Without giving details of their work in the paper! Just tell me the damn problem your tool solves regarding unix shell programming and why it's cool. Don't waste our time with "rehabilitating shell" in academia.
This is a weird claim. Every program's bahavior depends on its inputs in the general case. If you want to process static input you can do that using the shell. If your program should work on external inputs you cannot statically determine its output regardless of the language.
> The Unix shell is a powerful, ubiquitous, and reviled tool for managing computer systems.
Reviled? By those who use it often, it is more loved than reviled.
> The shell has been largely ignored by academia and industry.
Seems like a bit of an exaggeration, but it's true that corporations have a thing for creating custom proprietary software rather than providing shell-script-based solutions.
> While many replacement shells have been proposed, the Unix shell persists.
There is no single "the" Unix shell. bash is probably the most popular, but it isn't "the" shell, singular.
> Two recent threads of formal and practical research on the shell enable new approaches. We can help manage the shell’s essential shortcomings (dynamism, power, and abstruseness)
Again with "the" shell. Anyway, a shall should be dynamic and powerful; and I wouldn't say bash is _terribly_ abstruse. If you apply yourself, your code could actually be rather readable.
and address its inessential ones. Improving the shell holdsmuch promise for development, ops, and data processing.
i think its because people dont want to rtfm. if they would rtfm, they would learn most problems have a solution that is not insane like the one they are about to start developing. you know how often i see this:
cat somefile |grep thething instead of just: grep thething somefile
That's academia going pedantic and mystifying things right there. Strict POSIX compliance was never that important to most people. I never read the POSIX shell specification and never needed it. Writers of shells like bash or ksh probably did and apparently managed this "black magic" well enough to make their shells work well.
You can learn shell programming by yourself, from bash manpage and examples used in Linux systems. It's not rocket science.
Computer languages get adopted not because they have nice and simple spec, but despite not having a neat and simple spec. It's like with natural languages. There is no authority and no simple spec. And it's not an important problem.
I hope that this jump starts a bit more research into shell languages. Maybe the veneer of authority given by papers suggesting various improvements will help draw people to next-generation shells that take up those improvements and advertise that.
If you were not aware, PowerShell has inspired a whole generation of next-generation shells built around structured pipelines, and they're very exciting. My personal favorite is Elvish, which also draws much inspiration from Fish, a lovely 'old-fashioned' (text-only pipeline) shell with a strong emphasis on interactivity and ease of use. One thing that makes Elvish special besides its ambition to be a pleasant and useful general-purpose programming language is its portability: it's typically distributed as a single static binary, and it even has native support for Windows.
When I think of 'the future of the shell', I absolutely think of Elvish, namely with respect to:
* better portability
* high aims as a 'real' programming language
* built-in static checks where possible (for now just syntax, I think, but a more complete type system is on the roadmap)
* sophisticated and responsive interactive experience out of the box
* structured pipelines, structured pipelines, structured pipelines
Your description of Elvish is much better in that respect. I'd love more capable shell with modern advancements like types and static checking. However, I don't get why structured data over pipelines is useful. You can already exchange any kind of data over pipeline as string of bytes. Sender and receiver have to be aware of the format in any case.
The Elvish docs have a pretty good answer to this here[0], but I'll give my own answer, too.
There are a few things. Structured pipelines spare you from having to do escaping and transformations in order to preserve structure.[1] It's also a natural complement to a type system. Some other next-gen shells call their structured pipelines ‘typed pipelines’ instead.[2] Structured pipelines also provide a nice mechanism for separating the visual/textual representation of data from its structure, so you can ‘convert’ values between different display formats without doing any parsing, in a convenient, extensible way. PowerShell has this with Format-List[3] and Format-Table[4], for example. This kind of pretty-printing isn't implemented in Elvish yet, but Kurtis Rader helped me outline one possibility in an open GitHub issue.[5]
On some level, structured pipelines just mean that commands ‘native’ to the shell language (builtins and functions written in the shell language) all understand the same types or data structures. This means you can ingest data from some external format (JSON, CSV, TOML, a SQL result, whatever) and then use common tools. Nushell's first example for working with pipelines[6] hints at one of these benefits: you can use a single command, like `inc` to increment the value associated with a certain key in a map, with data that comes from any format, and `inc` doesn't have to know anything about data formats or parsing.
Another way to think about it is that it lets pipelines have the same kind of type checking as functions called with arguments. In a typed language, you get type checking for your arguments when you call a function like
(some_command arg0 (another_command arg1 arg 2))
You're right that if you are using a lot of external commands in a ‘raw’ way, you don't get to take much advantage of the structured pipeline. Imo PowerShell proves that in practice, if you have a big enough library ecosystem, you can leverage wrappers and ‘pure’ tools written in your shell language to get a pretty nice programming experience.
—
0: https://elv.sh/learn/unique-semantics.html#motivation
1: https://elv.sh/learn/effective-elvish.html#returning-values-...
2: https://murex.rocks/docs/user-guide/pipeline.html
3: https://docs.microsoft.com/en-us/powershell/module/microsoft...
4: https://docs.microsoft.com/en-us/powershell/module/microsoft...
5: https://github.com/elves/elvish/issues/1149#issuecomment-705...
6: https://www.nushell.sh/book/pipeline.html#basics
7: https://elv.sh/learn/effective-elvish.html#prefer-pipes-over...
I must admit I did not understand their message at the first quick reading.
Formal methods? Well, I'm not whether anybody tries to prove their installation scripts (or whatever people would script) correct.
Parallelization? Well, if you try to high-performance computing, is the shell really the place to start?
It's still early / unreleased but I'd like to add my own terminal [0] which lets the emulator have an understanding of user input -- rather than simply delegating to the shell.
I'm finding that the interactive pathways that open up are mostly-uncharted territory.
> Systemd uses its own vari-able expansion regime, slightly different from the shell’s...encouraging you to runsh -cif you have an actual pipelineto run. Dockerfiles and CI config files arealmostshell scripts,but there’s no convenient way to just execute theRUNcom-mands orscriptlists they contain. And few are the Vagrant-files that don’t call out to someprovision.shto set up de-pendencies. Giving up on the shell means that each ‘modern’system tool will have its own janky quasi-shell language:a decidedly worse situation.
But adding jet another POSIX shell which maybe does some parts better but in the end has still not a grate UX because it's a POSIX shell seems kinda pointless IMHO.
Pretty sure this is what Powershell does, but the UI just feels so damn unnatural.
Take a plain text table, convert it into an SQL table and run an SQL query:
» ps aux | select USER, count(*) GROUP BY USER
USER count(*)
_installcoordinationd 1
_locationd 4
_mdnsresponder 1
_netbios 1
_networkd 1
_nsurlsessiond 2
_reportmemoryexception 1
_softwareupdate 3
_spotlight 5
_timed 1
_usbmuxd 1
_windowserver 2
lmorg 349
root 134
» fid-list:
FID Parent Scope State Run Mode BG Out Pipe Err Pipe Command Parameters
590 0 0 Executing Normal no out err fid-list (subject to change)
» fid-list: | cat
["FID","Parent","Scope","State","RunMode","BG","OutPipe","ErrPipe","Command","Parameters"]
[615,0,0,"Executing","Normal",false,"out","err",{},"(subject to change) "]
[616,0,0,"Executing","Normal",false,"out","err",{},"cat"]
» fid-list: | format csv
FID,Parent,Scope,State,RunMode,BG,OutPipe,ErrPipe,Command,Parameters
703,0,0,Executing,Normal,false,out,err,map[],(subject to change)
704,0,0,Executing,Normal,false,out,err,map[],csv
» open https://api.github.com/repos/lmorg/murex/issues | foreach issue { printf "%2s: %s\n" $issue[number] $issue[title] }
348: Potential regression bug in `fg`
347: Version 2.2 Release
342: Install on Fedora 34 fails (issue with `go get` + `bzr`)
340: `append` and `prepend` should `ReadArrayWithType`
318: Establish a testing framework that can work against the compiled executable, sending keystrokes to it
316: struct elements should alter data type to a primitive
311: No autocompletions for `openagent` currently exist
310: Supprt for code blocks in not: `! { code }`
308: `tabulate` leaks a zero length string entry when used against `rsync --help`
Website: https://murex.rocks
In the end I settled on the design I have because it retains compatibility with the old world while enabling the features of the new world but it also behaves in a predictable way so it's (hopefully) easy for people to reason about. Powershell (and other languages in REPL like Python, LISP, etc) still exists for those who want a something that's ostensibly a programming environment first and a command line second and I think trying to compete with the excellent work there wouldn't be sensible given how mature those solutions already are. But for a lot of people, the majority of their command line usage is just chaining existing commands together and parsing text files. Often they want something terser than $LANG as a lot of command lines are read-once write-many and thus are happy to sacrifice a little in language features for the sake of command line productivity. This is the approach murex takes. Albeit murex does also try to retain readability despite being succint (which is probably the biggest failing of POSIX shells in the modern era).
What I've built is definitely not going to be everyone's preferred solution, that's for sure. But it works for me and its open source so hopefully others find it as useful as I do :)
On a related note, I wonder if and how a pipe could handle "format negotiation" between processes? I.e. is there a way for a CLI app to indicate it can consume and produce structured binary data? Then the piping layer could let compatible apps talk through an efficient protocol, and for anyone else, it would automatically drop to equivalent JSON (and then maybe binarize it back up, if the next thing in the pipeline can handle it).
You can already use BSON. The data is piped in whatever serialisation format it's typed as but the type information is also sent. Builtins then use generic APIs that wrap around STDIN et al which are aware of the underlying serialisation.
So the following works the same regardless of whether example.data is a JSON file, BSON, YAML, TOML or whatever else:
open example.data | foreach i { out "$i[name] lives at $i[address] }
> On a related note, I wonder if and how a pipe could handle "format negotiation" between processes? I.e. is there a way for a CLI app to indicate it can consume and produce structured binary data? Then the piping layer could let compatible apps talk through an efficient protocol, and for anyone else, it would automatically drop to equivalent JSON (and then maybe binarize it back up, if the next thing in the pipeline can handle it).
That isn't that far removed from how murex already works. Supported tools can use common APIs to convert the STDIN into memory structures, and similarly convert them back to their serialisation file formats. So if you have a tool like `cat` in your pipeline, they can use the pipe as a standard POSIX byte stream, but murex-aware software can treat the pipeline as structured data. The drawback of this is if you're reading from a POSIX pipe into a murex-command, you might need to add casting information (see below). But the benefit is you're not throwing away 40 years of CLI development:
# Using a POSIX tool to read the data file:
# casting is needed so `foreach` knows to iterate through a JSON object
cat example.json | cast json | foreach { ... }
# Using a murex tool to read the data file:
# no casting is needed because `open` passes that type information down the pipe
open example.json | foreach { ... }
Worst case scenario and you're dealing with gigabytes or more of data, then you'd a streamable format like jsonlines where each command in a pipeline can run concurrently without waiting for the file to EOF before processing it. In those situations most binary serialisations aren't well optimised to solve.
I like this approach:
https://github.com/kellyjonbrazil/jc
> CLI tool and python library that converts the output of popular command-line tools and file-types to JSON or Dictionaries. This allows piping of output to tools like jq and simplifying automation scripts
My ultra long term aim is to integrate the shell into a media rich terminal emulator so using the command line will have the speed and precision of a TUI, the power of an IDE but the rich content of a GUI.
I'm a loooong way off achieving that but the shell is already usable and has been my primary shell for ~4 years now. And I welcome any and all feedback on the current and any future builds.
At some point I'll publish a paper with my ideas.
I think HN is uniquely placed where it has a lot of trading topics as well as IT so it wasn't until my shell started trending on here that anyone had even noticed the naming conflict. But my shell had been around for a good few years before it started trending on here.
Could you help me differentiate it from Elvish in terms of features and design?
Think of it like writing/downloading type definitions for existing untyped code. With program descriptions like this shells could become smarter, generate warnings, abort before execution if types don’t match up, etc.
For something smarter that behaves more like a typed programming language, there's Perl, Python, etc.
Using `set -euo pipefail` and tools like Shellcheck help, but there's a limit.
https://sipb.mit.edu/doc/safe-shell/
^ Says the guy who forked a project an hour ago just to fix a couple errors, found a subshell and said, 'You're done, go no further' :)
That said I would agree as soon as something becomes complex or you need to do computation looking outside the shell is probably a good idea.
The huerestic that I use is
1. Do I need to do math? Move out of the shell.
2. Is the logic starting to get complicated enough that I need consider creating modules? Move out of the shell.
3. Is there enough code to fill more than one screen? Out of the shell I go.
4. Is this going to be something that needs robust error handling and that will be used frequently outside of me? Out of the shell.
I've found that as long as I don't hit one of those conditions hacking around in the shell is quite powerful and useful.
Still, I've found Rake and Make - as simple containers for shell rather useful and more flexible - newlines are sane, far less of this `\n` nonsense.
^ An odd complaint, but it just seems like the terminal or the environment should handle this without input from me.
Now if only I could do something about Make syntax requiring tabs...
Completely disagree. Just parsing a command-line takes a screen's worth of lines. Hell, just the usage info message for a script can take that much.
> 4. Is this going to be something that needs robust error handling and that will be used frequently outside of me? Out of the shell.
Disagree even more strongly. You're saying never to write shell scripts for use by many people. Actually, shell scripts make up a lot of a typical systems' /usr/bin and /usr/sbin - and that is fine and proper.