Is the "way forward" recombining and rehashing the terrible technologies everyone is familiar with already, indefinitely?
What do you mean by "UNIX shells suck"? I'm fairly sure most people would consider UNIX shells to be one of the bigger revolutions in computing.
But as soon as you want to do anything moderately interesting, it's better to grab a real programming language, because UNIX shells suck. Escaping failures, whitespace handling, almost-real-arrays, fail-and-continue by default, and many many other reasons you'll fail by accident one day is why they absolutely suck for almost anything past basics.
In a shell, any bareword is either one of a few built-ins or an external program. Using a bareword which doesn't correspond to anything isn't a syntax error, because the parser can't know what programs you have installed from moment to moment. That is the biggest difference between a shell CLI and a programming language REPL.
This is, in my opinion, the beauty of UNIX shells. They're simply enough to use like DOS but powerful enough to use like a REPL. It's not perfect; far from it. But few other tools support such a wide demographic.
The shell languages are not much better.
There are some useful concepts, but in general Unix shells are used despite their user unfriendliness.
The shells of the various Lisp Machines were quite different. The Symbolics shell, later called Dynamic Lisp Listener was quite nice on the GUI side and the management of commands, completions, defaults, interactive help, etc..
See for example: https://www.youtube.com/watch?v=o4-YnLpLgtk
The interactive help of that Lisp Machine OS is quite a step up from what any typical shell offers. Though it was mainly developed for single user machines with powerful GUIs.
The problems of that approach: it wasn't very sophisticated on the text terminal - actually it was quite bad. The whole UI mostly assumed a GUI. For development one needed an extended Common Lisp (or Zetalisp), which is a bit too complex for many users.
See also a video I made long ago about the user interface of Symbolics Genera, the operating system of the Symbolics Lisp Machine line of personal workstations.
They're not that bad (GNU coreutils is generally pretty good) and you usually remember the edge cases fairly quickly (eg `dd`). Sadly you get inconsistencies in all coding frameworks, whether it's semantic, function names in the core libraries or whatever.
UNIX shells do have a lot of hidden gotchya's which can make life "interesting" (read: "oh fuck oh fuck oh fuck" lol). But the power of being able to use reusable blocks of any language through pipelining (and to a lesser extent, exit codes) is genius. It means one can mix and match LISP, Java, Perl, Python, C++, Go all inside one shell script.
I do understand the hate towards UNIX shells. There are a lot of faults and a lot of times I I'd be halfway through writing a Bash script and then be wondering if I should have just written it in Perl or Go instead. But no tool is perfect and pragmatically I've found shells to be far more productive than anything I've ever attempted to replace it with. Which is the real crux of why we use these tools. But like anything in IT, this is just my personal preference. Your mileage may vary.
Generally this is all awful and low-level. Just see the UI interface difference between 'dd' and 'Copy File' on a Lisp Machine. The UI is worlds away.
> UNIX shells do have a lot of hidden gotchya's which can make life "interesting" (read: "oh fuck oh fuck oh fuck" lol). But the power of being able to use reusable blocks of any language through pipelining (and to a lesser extent, exit codes) is genius. It means one can mix and match LISP, Java, Perl, Python, C++, Go all inside one shell script.
You can do that on a Lisp Machine, too. With the difference that no pipelining of text is necessary. Just reuse the objects. The data is all fully object-oriented and self identifying.
Note that I'm using Unix and other OS variants since the 80s and I'm fully aware of command line UIs from VMS, SUN OS, Cisco, IBM AIX, OSX, GNU, Plan9, Oberon OS, and various others.
> I do understand the hate towards UNIX shells.
It's not hate. Most Unix shells are just fully dumb. Many people like to use primitive text based UIs with lots of corner cases, which makes them look intelligent for remembering obscure commands, obscure options without real UI help.
Take cp.
The shell does not know the various options the command takes. The shell does not know what the types and the syntax of the options is. The shell does not know which options can be combined and which not. It can't prompt for shell options. It can't check the command syntax before calling it. It can't provide any help when the syntax is wrong. It can't deal with errors during command execution. There is no in-context help. It can't reusing prior commands other that just editing them on a textual base. The output of the command is just text and not structured data. There are really really zillions of problems.
There have been attempts to address this putting different user interfaces on top. For example IBM provided an extensive menu based administration tool for AIX.
> But no tool is perfect and pragmatically I've found shells to be far more productive than anything I've ever attempted to replace it with. Which is the real crux of why we use these tools. But like anything in IT, this is just my personal preference. Your mileage may vary.
Many people have found shell productive. That's why there is a zillion of different shells. You can even use Lisp-based shells like scsh and esh (http://www.serpentine.com/blog/2007/02/16/esh-the-fabulous-f...) or GNU Emacs.
But for most part all these attempts stay in that little box and don't escape the general problems.
> Generally this is all awful and low-level. Just see the UI interface difference between 'dd' and 'Copy File' on a Lisp Machine. The UI is worlds away.
Well yeah, I did already example `dd` as an inconsistency. :)
> You can do that on a Lisp Machine, too. With the difference that no pipelining of text is necessary. Just reuse the objects. The data is all fully object-oriented and self identifying.
Indeed you can. As you can also with DOS, Powershell and so forth. I wasn't suggesting that UNIX shells were unique (though I can see how it might read that way), but that it was UNIX shells which pioneered that concept. For all their faults and the technology that might have superseded it: the idea of pipelining reusable blocks of code was a genius idea for its era.
Powershell also supports passing objects like Lisp does. Personally I prefer the dumb approach; however in all other aspects of programming I do prefer strongly typed languages. This is just personal preference.
> The shell does not know the various options the command takes. The shell does not know what the types and the syntax of the options is. The shell does not know which options can be combined and which not. It can't prompt for shell options. It can't check the command syntax before calling it. It can't provide any help when the syntax is wrong. It can't deal with errors during command execution. There is no in-context help. It can't reusing prior commands other that just editing them on a textual base. The output of the command is just text and not structured data. There are really really zillions of problems.
This part isn't accurate. The original Bourne shell cannot but bash, zsh and fish can all do all of the above. Albeit sometimes (particularly with Bash) you need to install additional helper routines that aren't always shipped / configured with the default package. I believe csh also supports most if not all of the above too.
I'm sure lisp does it better, but I was never arguing that UNIX shells are better Lisp to begin with. Just that I believe Bash et al to have a lower barrier of entry than Lisp. I think on that specific point we might have to agree to disagree - but I don't see many non-programmers using Lisp and this is why I think Bash has a lower barrier of entry (to go back to my original point).
I can completely understand and relate to why you enjoy working inside a Lisp REPL shell though.
Lisp does nothing. It is a programming language.
> Just that I believe Bash et al to have a lower barrier of entry than Lisp
I doubt that, given how horrible Bash as a language and as a shell is. There is no reason why there can't be more sane command systems and better shell languages.
See the zsh documentation on completion:
http://zsh.sourceforge.net/Guide/zshguide06.html
This is all totally over the head of the average user.
_perforce_revisions() {
local rline match mbegin mend pfx
local -a rl
pfx=${${(Q)PREFIX}%%\#*}
compset -P '*\#'
# Numerical revision numbers, possibly with text.
if [[ -z $PREFIX || $PREFIX = <-> ]]; then
# always allowed (same as none)
rl=($rl 0)
_call_program filelog p4 filelog \$pfx 2>/dev/null |
while read rline; do
if [[ $rline = (#b)'... #'(<->)*\'(*)\' ]]; then
rl=($l "${match[1]}:${match[2]}")
fi
done
fi
# Non-numerical (special) revision names.
if [[ -z $PREFIX || $PREFIX != <-> ]]; then
rl=($rl 'head:head revision' 'none:empty revision'
'have:current synced revision')
fi
_describe -t revisions 'revision' rl
}
> but I was never arguing that UNIX shells are better Lisp to begin with
I was not talking about Lisp. I was talking about software. One could write much better command shells in C than what Unix shells offers. I understand that certain people find Unix like shells attractive. From a general user interface perspective they are horrible.
In both cases, for small uses, you can stretch your shell or REPL or programming language over to the other case, but for long term use it's just not practical. After all, in theory many languages have REPLs that could have long since replaced shell, but only a handful of very dedicated people actually replace their shell with a REPL for some language. I've tried, and I can't do it, even with shell support libraries in the relevant language.
The suckiness of UNIX shells is mitigated if you view them as "the tool optimized for interactive usage". And once you let them be that, they really aren't so bad. It's only if you try to force them to be programming languages, especially past a couple dozen lines or when they try to do something other than just run lots of commands pretty much blindly, that they become really bad.
Also complicated quoting is a pain, but as a percentage of the number of commands I run in my shell, they aren't that large, they just loom large in my memory. Run history and really look at what you're doing day in, day out with your shell. Unless you've got a really crazy usecase, there's a lot of things like "ls" followed by a "cd dir" and such. It's part of why I can't get into REPL replacement; replacing "cd tmp" with even "cd('tmp')" is, percentage-wise, a huge increase in keystrokes. (And don't forget, the space is much cheaper than left paren; the space bar is huge and I basically have an entire digit dedicated to it, whereas left paren is two not-on-home keys at the same time.)
I had been juggling the idea of using a modified REPL for shell work, but I haven't written much code. However, someone has written such a shell, namely Avesh[0]. Check out the Reddit comments as well [1].
I haven't used it yet, but the examples seem to fit what I would want in a Common Lisp shell: * Simple sh-like commands with flags (e.g. ls -o, grep -i) * Handling commands and piping as native CL functions (e.g. | (| ls (grep c))(grep r) ).
[0] = https://gitlab.com/ralt/avesh [1] = https://www.reddit.com/r/lisp/comments/48r70b/avesh_the_supe...
Everything's an object. Instead of passing lines of text, you're passing objects, with properties, types, etc. This becomes important because of
Library files are first class citizens. Because the shell is string based, you have to wrap everything up in functions, creating helpers in order to use any sort of function. Powershell makes this trivial, such that you can use functions you wrote elsewhere in your scripts as if they were builtins. Hell, it's trivial to write a webserver that forks multiple processes in powershell.
Yes, the unix shell was revolutionary when it was released, but it's been hamstrung by tradition. The shell is long overdue for letting in more interesting abilties.
Hybrids like ipython shell are quite interesting for personal use though.
I get why Powershell is great for sysadmins (and especially on Windows, where so many system-level utils weren't built with the idea that you'd pipe their output in mind), but for me, it's a step back.
Having said that, I can't vouch for your scripts, but my 10 line shell scripts are unlikely to handle 'edge' cases such as spaces, quotes and backticks in file names.
Also, "incomprehensible to anyone who doesn't know .net" may be true, but your average shell script that builds on tools such as sort, grep, ls, cut, cat, sed, awk, etc. isn't that particularly comprehensible to people who don't know UNIX, either.
Finally, I think it is likely that people who grow up on Powershell feel differently about this.
Absolutely. The point is, "you can use familiar .NET functions and everything is an object" is not something that makes the whole thing easier if you're not familiar with .NET and don't want objects in your shell scripts :-).
Powershell and Unix shell have vastly different descents, and it shows. It's more or less a historical accident that we're conflating them nowadays.
Like everything on the Internet, lack of success with a technology should be taken to mean a failure of that technology as much as a failure of the programmer attempting to use it. It may well have been a problem above the application layer.
If I'm spending twice the time typing on average, even more advanced features won't win me over when I get to use them occasionally (while, again, typing way more on average). It should be saving you time, in the end.
Things are better now with builtin tests, but god help you if you need to use /bin/[ and you end up with idioms like 'if ["x$foo" = "x$bar"]' so that bad things don't happen if $foo starts with a dash, or either variable is empty.
It gets even more fun when you look at the amount of cruft around terminal emulators themselves, and the support for legacy terminals in the command line. Why do modern versions of OSX still ship with /usr/lib/libtermcap.dylib?
Don't get me wrong -- UNIX shells are amazingly powerful tools, but they're just stuck in the 70s and we can do much better today.
I feel like the interface for these commands is exactly what you'd want from a collection of executables that do only one thing.
As for shell scripting, you don't have to use bash/sh/whatever. You could write your script or executable in any language that will let you call system commands, so it's really a non-issue if you absolutely can't stand the syntax. I write plenty of bash scripts and I do feel your pain, they're not usually very pleasant if you have to go beyond trivial use cases- but it's also (generally) not impossible.
Hadn't thought of chaining that with a csv-to-json workflow, that's a neat idea -- but if that's the easiest way out, that's clearly a symptom of the problem I'm describing, isn't it?
One of the things I've been trying to figure out for a while is a way to build a trivial adapter for the csv-to-json | jq pipelines I've been building, something sufficiently easy to use that I could bake it into a |-equivalent extension to something like zsh.
The security ceremony required around PS is also an obstacle to getting started. And of course Microsoft have missed the opportunity to fix the path separator.
I admit that Bourne shell is not great for anything over a few lines, and find it better to drop into perl or python where filename-separator issues are mostly eliminated.
Sounds like you are looking for Tee-Object[0].
"Saves command output in a file or variable and also sends it down the pipeline."
And of course Microsoft have missed the opportunity to fix the path separator.
Windows supports both path separators in pretty much all cases (open files, changing directories, etc)
PS C:\> d:
PS D:\> cd /projects/node_modules/immutable/node_modules/uglify-js/node_modules
PS D:\projects\node_modules\immutable\node_modules\uglify-js\node_modules>
I guess this depends on what you want out of the system. Some options are:
"format-table -auto" which provides a nicely formatted table view of the data
"format-list" which provides each property on a separate line with a blank line between objects
"format-custom" which allows you to create your own views of objects (with a very readable default), which has ultimate flexibility
"converto-csv" which output the object as a CSV (or with custom delimiters such as TSV)
"converto-html" which outputs the object as an html table (with somewhat customizable markup)
"convertto-xml" similar to the above
But really, if it's possible, then it makes sense to use the object pipeline until you're done (so that different parts of the pipeline can use different parts of the object). Then spitting out CSV, TSV, XML, etc should get interop with basically every other tool that's going to process text in some way.
[0] - https://technet.microsoft.com/en-us/library/hh849937.aspx
The output of tee-object can't easily be read back in to a new pipeline. I suppose we're supposed to stash it in a variable which will be good enough for 99% of cases.
There's still a lot of room for improvement.
1. Win+R -> cmd enter is a lot faster than Win+R -> powershell enter
2. cmd starts faster than powershell
3. I seldom require all the power from powershell
npm install cash-global -g