Zsh 5.9
zsh.sourceforge.io
zsh.sourceforge.io
ls **/*.d
gives me all files, recursively, that end in .d. That's available in bash too if you `shopt -s starglob`.But it also gives all directories, so if I do eg
grep pattern **/*.d
then I get lots of error messages about trying to grep on directories (if I happen to have directories ending in .d). With zsh, I can do grep pattern **/*.d(.)
and it will restrict to just files. I use this pretty often, and it saves me from many `find | xargs` commands. I pretty much only use `find` when there are too many matches for a single command.My muscle memory types
fgrep -l SOMESTRING **/*(.)
multiple times per day. That'll list out all files containing the literal SOMESTRING anywhere in or below my current directory. I'm sure I could be using one of the fancy grep replacements, and I have in the past, but this works Good Enough and only requires zsh to be installed, so it's what I have held onto.Although for typical "shell-scripting" I still use BASH for the most-part where scripting languages like Python aren't as suitable, so my scripts can be used on other peoples systems.
I am admittedly a not very good bash'er, and this week at work I whipped up my first "complicated" bash script, but ended up rewriting it in Python. It might have just been the case that the IO bound nature of my task fell right into Python's strong suits (async IO), but it had me wondering what scenarios a bash script would be better.
Essentially I had to:
* iterate over a list of hundreds of thousands of files
* make an api call via aws cli
* take result and process it through a few shell utilities (`date` and `touch`) to then update the timestamps on the files.
I initially wrote it in bash which spawned ~16 background workers (threads/processes?) via `&` and blocking via `wait -n`. This "worked" but was pretty slow as the threads were thrashing checking for responses. Doing anything more than 16 caused my computer to crawl lol.
I then rewrote it in Python with async IO and the async subprocess API (to run shell commands) and it was an order of magnitude faster.
I wish I was better at bash, but maybe I just haven't spent enough time with it. Doing this task made me feel like I could pretty much doing most things in Python if I need a non-trivial script.
If I write a Bash script, I can be reasonably certain it will run on any unixoid system that came out in the last years. With Python, I am now in version hell (it's ridiculous how many Python2-first-servers I still find), I'll run the risk of includes suddenly becoming incompatible or buggy.
Can Bash do everything that Python can? Almost certainly not. But it is available, it is relatively simple (and almost minimalistic), and it forces you to learn more about standard unix tools.
Your specific use case ... well, I think that's pretty unusual - I'd wager if you had avoided the parallel processing and done it in a more linear way, you've had a better time.
> if you had avoided the parallel processing and done it in a more linear way, you've had a better time.
I had to process ~500,000 files, and the aws api call was on average ~1 second, so it would have been a significantly longer time to process linearly. For example the bash version I whipped up processed ~30k files in 2.5 hours, while the python version did 30k files in ~20 minutes.
But yes I agree, if I didn't have to do such a large volume at once the bash version would have been just fine.
Except for, like, all the (free) BSDs and Illumos. What “unixoid” systems were you referring to, exactly, other than Linux?
Don’t get me wrong, I accept that there are legitimate use cases for bash, but I really can’t help but feel that anything longer than 3 lines (including #!) is better off in anything*.
The shell’s parallel processing capabilities shine when you want consecutive stream transformation steps to run in parallel (note that you can pipe to and from loops, over multiple descriptors if necessary). When your task can be structured like that, it will often take as much time to write a parallel implementation in shell as it will a serial one in Python (etc.). On the other hand, running multiple iterations of the same step in parallel is, as you’ve seen, awkward.
If the only thing you actually need to parallelize is HTTP requests, you might find piping things into aria2c[1] or the like to generate temporary files with responses and then processing those serially (best if your tool can tell you on stdout when a job is done, otherwise inotifywait and its ilk[2] may help if you’re uncomfortable just hammering the filesystem). When that is not enough, you could try GNU Parallel[3] or, if all else fails, implementing a jobserver in the style of GNU Make[4], which you may find more pleasant to do in shell than your current design. Generating a Makefile and then letting Make manage the worker pool (and call out to a separate shell script for each job) is also a possibility.
But at that point (in particular if you’re not I/O-bound) it might be best to just not do it in shell[5].
[1] https://aria2.github.io/manual/en/html/aria2c.html
[2] https://jvns.ca/blog/2020/06/28/entr/
[3] https://www.gnu.org/software/parallel/
[4] https://www.gnu.org/software/make/manual/html_node/POSIX-Job...
[5] https://sanctum.geek.nz/etc/emperor-sh-and-the-traveller.txt
Yes, you can write a run() function that handles everything, and there are some 90% solutions in subprocess now, but the defaults are not as simple as a modern shell on this dimension.
Of course, if you need to do any string manipulation, advanced conditionals, or math, Python quickly pulls ahead.
Said functionality for zsh:
So I moved to zsh, enabled two plugins and borrowed a config. I have 99 percent of what I like about fish. It's quite awesome.
If anyone is looking to try out zsh, please please please don't start with oh-my-zsh. I actually went zsh-fish-zsh. My first zsh attempt being with oh-my-zsh.
autoload -Uz compinit; compinit
setopt menu_complete
zstyle ':completion:*' menu selectI had to turn off some of the Zsh-specific brackets and quoting features to make it more bash-like because of the tools I use which make it kinda wonky to use with Zsh but otherwise it’s been absolutely fantastic. Remoting into our systems that only have bash feels absolutely prehistoric.
It doesn't make a lot of a difference. For me the major selling point of zsh is that it has two forms of process subsitution: regular <(...) as in bash, based on sockets, and =(...), which creates temporary files. This allows you to do some neat things that are not possible or comfortable in bash.
Sometimes there are alternatives or workarounds, but the fact is that using fish does come with that downside. Doesn't come up often, but it is annoying when it does come up.
I like fish btw, just saying that I disagree with your "no reason to continue using zsh" comment. I use zsh because I was fed up with random things breaking due to not having a posix shell. And zsh can be customized to have most of the niceties of fish anyway. But it is pretty cool how fish comes with so many niceties out of the box!
Less than a year ago, I have tried zsh, without a definite intention to switch to it, but eventually I ended always using zsh for interactive use, while continuing to use bash for scripts.
I have benchmarked some scripts with both zsh and bash, but I have not seen any significant difference in speed.
While there are a few scripting features originally introduced by zsh (most of the brace expansion variants), bash has also incorporated them many years ago. Besides those, both bash and zsh implement the ksh93 features, so there is little reason to prefer one or the other. Therefore I have stuck to bash for scripts.
It is hard to say exactly why now I prefer zsh for interactive use, because there is no killer feature, but mostly various small details.
For some reason, zsh seems more responsive, when interactive. I rely a lot on autocompletion and the default autocompletion rules in zsh are more powerful and also the zsh autocompletion seems faster.
While in most cases I prefer the way zsh handles autocompletion, there are times when zsh annoys me by trying to be too smart, e.g. it will never autocomplete "mkdir" with the name of an existing file. I need that frequently, when I want to create a subdirectory in which to move a bunch of related files, with very long names that I do not want to type, but I want to edit slightly to make a name for the new subdirectory, so I have to use a workaround like autocompleting a "ls", then editing it into a "mkdir", while that was not needed in bash.
In zsh I use a much more complex shell prompt that I had used in bash, which displays a lot of useful information in a format that avoids confusion with the command lines or with the text output of the executed commands.
After switching to it in zsh, I have discovered that such a complex shell prompt is also possible in bash. Nevertheless, after using it in bash I have discovered some bash bugs. The too complex shell prompt confused bash and the text cursor became desynchronized with the correct position in certain contexts, so I had to revert in bash to a simple shell prompt.
I think that you should give zsh a try for interactive use. After using it for some days you will see if you prefer it to bash or not.
Now I prefer the interactive zsh, but there are no large differences, bash and zsh are similar and mostly equivalent, because whenever one was improved somehow, the other also copied that feature soon.
I had completely forgotten, but when I have tried for the first time to use zsh as a bash user, I have spent some ten minutes of great confusion, until discovering that I have to add to my ".zshrc" the lines:
bindkey '^[[H' beginning-of-line
bindkey '^[[F' end-of-line
bindkey '^[[3~' delete-char-or-list
in order to have the line editing keys with which I was accustomed.
PS1="%# "
RPS1="%40<...<%~"
This way I have current directory information without eating precious space.Other than that, I don't really use much zsh-specific functionality.
setopt transient_rpromptIt displays configured text at right side of your terminal. My example displays current directory truncated to 40 characters.
> and what happens if the text you're typing runs into it?
It disappears.
The "command1 =(command2)" syntax, which runs "command2", pipes its output to a new temporary file, runs "command1" with that file's name as an argument, and then removes the temporary file, all in one operation. It's like "<(command2)", which is in Bash and Zsh, but allows "command1" to perform a seek on the file because it's not a pipe.
Redirections like ">file" where there's no command implicitly use cat, so ">file" just pipes STDIN to the file. Same for "<file", which pipes file to STDOUT.
"command &>/dev/null" is shorthand for "command >/dev/null 1>&2".
Floating-point arithmetic built-in (Bash has integer arithmetic only), including C math functions like sin, log, floor.
Implicit insertion of "tee" where you obviously want it; for example, "command >file1 >file2" is the same as "command | tee file1 > file2"
You don't need to quote variable expansions like $1 (except arrays). You still have to do so for command expansions $(command
More expressive pattern syntax, including backreferences, approximate matching (using Damerau-Levenshtein distance), and flags/qualifiers to limit searching to specific kinds of file. E.g. "ls *(-@)" lists all broken symbolic links.
The string =foo is expanded to the path to the "foo" command in $PATH, which allows you to do, for example, "vim =myscript" without needing to remember where "myscript" is exactly.
Lots of parameter expansion flags for various string operators (search/replace, sorting, case translation, splitting and joining, ...).
POSIX extended regex and optional PCRE support for pattern matching.
zle, which does a similar job to readline in bash, but with an enormous number of useful features built-in.
Generalised tilde expansion: ~ is the current user's $HOME, but ~john is the user john's home directory. You can make up your own using "hash -d foo=/path/too/foo"; now "~foo" refers to "/path/to/foo".
Dynamic version of the above: ~[string] uses the function zsh_directory_name to expand "string" into any path; see the manual "zshexpn(1) § Filename Generation § Dynamic named directories".
Easily create more explicit pipes using file descriptors:
# open
exec {fd}>file
command1 >&$fd
command2 >&$fd
#...
# close
exec {fd}>&-
Additional concise and useful syntax for loops, function definitions, etc.: while (( x < 5 )) { echo $x; let x++ } # more obvious than "
repeat 5 echo hi # does what it says on the tin
myfunc() echo hi # declares a function
for ((x = 0; x++; x < 5)) { echo $i } # C-style for-loop
The equivalent of "try-finally" in some programming languages: {
# do some stuff which might error
} always {
# do some important cleanup
}
This construct also enables special support for another feature: actual exceptions with throw/catch.Very flexible prompt expansion: conditionals using %(condition.true.false). Optionally, command expansion "$()" can be enabled in prompts.
All kinds of extra optional modules which can be loaded using "zmodload", for manipulating xattrs, POSIX Capabilities, TCP sockets, an FTP client!?; also, built-in plugin functions which can be loaded with "autoload": a full calendar application, an interactive calculator, colours, regex replace, tetris!
> Zsh was once accused of not being as complete as Emacs, because it lacked a Tetris game. This function was written to refute this vicious slander.
autoload -U tetriscurses
tetriscurses
Ability to set the value of argv[0] received by a command, rather than just the name it was invoked by, e.g. "ARGV0=ls busybox"Options to emulate the behaviour of other shells: pure POSIX sh, ksh, etc.?
Sorry, that was perhaps a bit more than "some of my favourites". It really does have a lot of stuff.
zsh never clicked for me though, until I used oh-my-zsh, which allows for sharing custom extensions from folks. And it’s great, and superior to bash.
I used to not care, but these little improvements are worth it.
I recommend Oh My Zsh with the zsh-autosuggestions [1] and zsh-syntax-highlighting [2] plugins.
[1] https://github.com/zsh-users/zsh-autosuggestions
[2] https://github.com/zsh-users/zsh-syntax-highlighting
And my custom theme ;) https://github.com/whyboris/dotfiles/blob/main/yboris.zsh-th...
https://github.com/zsh-users/zsh-syntax-highlighting/blob/ma...
Looks like you need to clone the .git repository into your oh-my-zsh directory
zsh + oh-my-zsh + powerline10k on iterm2
alternative, less fancy, but nice if you want something simple yet decent:
zim on alacritty
1. it being the new Mac default
2. oh-my-zsh
And if the second is important, I wonder how many people have never tried anything similar for bash, such as https://github.com/ohmybash/oh-my-bash (never used it myself, but it has stars!)
However, on my personal machine and at my previous employer, I used a reasonably tweaked oh-my-bash setup and I can't say that I notice much of a daily quality-of-life difference.
Also remember that productivity is difficult to attribute to a single factor. Just having a more pleasant shell helps staying focused and motivated, even though the settings themselves do not improve efficiency directly.