Shell is required to be basically everywhere. It's part of POSIX. It's also standardized. Shell scripts written 30 years ago still work, and will probably work 30 years from now too. Shell is probably on your TV.
Not everyone has Python. The most common version in the field is Python2, but that's officially obsolete. Python3 is in many places, but not everywhere. The two Pythons are basically incompatible unless you're willing to put in an effort. You have to work to make Python scripts compatible with both versions, especially if you can only use built-ins, and the results are way clunkier than that 2-line shell script you're trying to replace.
Python is a terrible choice if your goal is a relatively short script that calls other programs. You can call out to other programs in Python, but it's much clunkier and more painful to maintain. And while it often doesn't matter (because neither are speedy), CPython 3 takes more than 50x more time to start than dash and 27x time to start than bash (see: https://github.com/bdrung/startup-time , which shows Python3 3.6.4 at 197.79 ms, Bash 4.4.12 at 7.31 ms, and dash 0.5.8 at 2.81 ms). That speed is helpful when you want to have a "simple script in a loop".
Python is much better than shell when you start needing more sophisticated data types, libraries, modularity, etc. But the typical use for a shell script doesn't need any of that. If you need that, shell is a terrible tool, because it's the wrong tool for the job, not because it's never useful.
If you're writing shell, use shellcheck. Once you do that, your error likelihood goes way down. Many of the reports of problems writing shell come from a time when shellcheck didn't exist.
POSIX shell being fixed forever though also means it will /never/ get "fixed".
Shell's silent-failure-by-default, implicit-by-default, principle-of-most-surprise semantics will always be there, and they are /bad/.
I have read lots of shell scripts, by many people, from the experienced UNIX greybeard to to the novice web developer, and they were almost all flawed. Pointing out the flaws, the devlopers of each level were surprised by them. It is close to impossible to write correct shell script, no matter the level of expertise, because the language offers neither semantics nor tooling that foster correctness.
Forgetting `set -e`, forgetting that it gets turned back off in subshells, quoting mistakes, pipefail semantics, you have to have made many mistakes to even /discover/ that these pitfalls exist (or use shellcheck, which will find some, but certainly not all of them).
Shell is a language that looks simple on the surface, but has 1000s of exceptions you have to master to write even minimal non-bugged scripts.
The lack of any form of reasonable data structures and strings being the prevalent data types also do not help. For example, NixOS's linker wrapper script (written in bash) was accidentally quadratic because of quadratic append-to-strings (https://github.com/NixOS/nixpkgs/issues/27609); and that was despite NixOS having many competent bash programmers. When scripts grow, these things happen.(Note that shellcheck does /not/ point out quadratic string appends; it can only find obvious mistakes and this is hard to detect in general). Fixing it was very difficult, and was not understood by many. If you have a language like Python that offers reasonable set/dict data structures that people understand and use intuitively, this is trivially avoided.
Finally, POSIX does not help much when the main thing shell does is calling other programs, and there are no guarantees on what the APIs for those programs are. For example, I have encountered many shell scripts that themselves are technically "portable", but they use arguments to e.g. `rm` that do not work on e.g. busybox, thus crashing the boot processe and resulting in hours of time lost for hundreds of people.
By not using shell, we can support that the next SOMETHING-standard will NOT mandate shell, but something better.
A Python script is not the answer when I'm trying to do something quick and dirty at the shell.
The beauty of shell scripting is that it evolves seamlessly from trying to solve simple problems at my command prompt. I pipe a couple of things together, and then I realize I could use a loop and a few conditionals, and suddenly it makes sense to store this in file in case I want to do this again. Boom, program done.
It now works on every computer I own.
I love Python. I use it almost every day. But I'll be damned if I'm going to go rewrite every shell snippet I hack together into Python just because it exists.
Different problems, different solutions.
This kind of automation adds a lot of value for low marginal effort and probably explains a lot of the short scripts I have laying around in directories.
The major differentiator is usually "shelling out" in Python kind of sucks. It's verbose, output collection and error handling suck, and escaping can be miserable. I often will reimplement things in pure Python if I have the time.
A recent example was I needed to tar+split large files. `tar cf - -C / $filename | split --bytes ${size}MB --verbose - $tmpfile.` My pure-Python implementation used the Tarfile library and I wrote a custom file-like object to split it--at least 50 lines. Both methods have pros/cons, but I had both implementations handy depending on the context I needed it in.
Another recent example was something I wrote to merge multiple video file "parts" into a single video using ffmpeg. It's 4 lines of shell script and would at least be 3x as long to write in Python.
The rule of thumb I have is if it's longer than 20 lines reevaluate if a shell scripting language is appropriate and I haven't really seen a need to change that in ~15 years I've used it.
Since I did this at work it belongs to my employer, so I can't currently publish it freely, but it's not part of our core product so they may be open to publishing it under MIT licence or similar at some stage.
So it can be done, and has been done, but I guess most people are sufficiently happy with Powershell, Python, etc to not bother bridging the gap for C# too.
I think it's more incongruence between the languages. os.system() will technically call a command. subprocess is a big step up from popen2, but Perl was much more streamlined and terse. That also applies to one-liners when you're in a shell.
With Python you just have differences in a bunch of things; error handling? I often have to look up error codes, catch the exception then check for error codes. Pipe data in? Test for sys.stdin.isatty() then read from sys.stdin or fallback to sys.argv--it's not obscure, just not particularly Pythonic. The list goes on and on. It will probably take ~20 lines to properly deal with shelling out and if you do things wrong you could deadlock[1]. On the plus side, discouraging shelling out means your code is more portable =P
[1] https://docs.python.org/3.8/library/subprocess.html#subproce...
If we understand "box" as "environment", in general, and not literally a physical or virtual machine...
Not really. E.g. see: every clean Ubuntu Docker container. They don't have Python, nor should have to include it by default.
And if you are going to do something like this:
apt-get update && apt-get install -y stuff
wget https://file.tgz
tar xf file.tgz
cd file
make
make install
value="$(cat result | grep something)"
mvn clean install -Dapp.value="$value"
I recently did the opposite of what everybody seems to be doing: moved an 800-line Python script, which you had to read carefully to understand what it was doing, to a 150-line Bash script that you only have to look at, because the important things to know is to see what other, external programs are being called. Just the apt-get stuff in Python was a mess with the poorly documented Python API for Apt, and some 50 lines to just do the same of an apt-get install... effectively wasted mental cycles.
(EDIT: formatting)
There seems to be something very wrong with this expression. Why would one need API for apt? It's a command-line tool.
Bash, and Bash-like shells, are literally everywhere. I have to be wary about what Python 3 features I use, and if there will even be a Python interpreter available. My OpenWRT router has a Bash shell, but I don't care to install and maintain other interpreters or runtimes on an embedded platform.
Even job management and parallelism are easier in Bash and GNU Parallel.
It really feels like piping and easy process invocation and compile-time directory awareness wouldn't be massively onerous to add to an existing full-featured programming language so you wouldn't have to sacrifice a good type system and powerful syntax when you want to do scripty things.
Why do you think it is not a programming language?
PowerShell meets both requirements as it's tuned for the Windows environment, but can still play nicely with the rest of the world.
Of course, now that WSL is a thing, it's a real question why someone would want to use PS unless they had no other options.
https://www.heavybit.com/library/podcasts/to-be-continuous/e...
I use it because I only have to learn one tool, and not a ton of different ones with different UX.
The 'language' is sane, at least to me, as it's object oriented doesn't revolve around parsing strings.
edit: your comment inspired me so I installed PowerShell via Snap and am going to give it a go on Linux.
That's the good part. The bad part is that it's easily one of the wartiest languages I've ever used, especially as a young language it's really got an inexcusable amount of legacy problems.
A colleague of mine (who also uses it heavily) says "two Googles per line".
Just to get sane parameter and variable checking you have to throw up a bunch of attributes and set a bunch of flags.
Be sure to set up strict mode, use cmdletbinding on your parameter blocks, and set ErrorActionPreference to "stop" in every script you write.
That and the quality of many of the first-party official PowerShell modules is appallingly low by Microsoft standards. The good side is that the .NET framework is accessible so whenever the PowerShell module is failing you, you can drop down into leveraging raw c#-style assemblies, which are much higher quality but they speak PowerShell with a very thick accent.
But I'm using it on Windows, YMMV on Linux.
I'm infatuated with the concept, but the implementation leaves a lot to be desired.
Check out dshell: https://pub.dev/packages/dshell
Next Generation Shell - mostly there. If something of the above is missing - it's a bug and you are welcome to open an issue. I'm the author.
Just curious, but how would one do process substitution with NGS?
There's a reason AT&T used to run ads that showed their secretaries, managers, and so on using and writing shell scripts and there's never been a Python ad claiming that just anyone could write it.
I reckon that less than 0.1% of the population can read shell scripts.
PR's welcome. ;)
Python makes sense for scripts that need good argument parsing, or complicated intermediate input processing. But it's pretty annoying to get a shell pipeline working in Python. `foo | bar | baz` is about 15 characters in bash.
Here's an example. I use `llbranch` all the time: https://github.com/shawwn/scrap/blob/master/llbranch
It lists all branches in a project. It's nicely colorized. What would the equivalent Python be? Who knows, but it'd be much longer.
A shell script that consists of a one-line exec isn't what people concerned about.
FWIW I'm a 99.9% POSIX masochist (probably closer to 4N)
I have maybe 3 standalone shell scripts in my PATH, despite writing thousands of lines of shell.
What moved me to putting them in a bin directory was that it's just easier to edit a standalone script.
To my mind, the parent post was referring to lengthier, more complicated scripts. Speaking as one who wrote and routinely maintained such scripts at one point, I cannot agree more with the sentiment that most if not all of them could and should have been written in Python or Ruby or some other scripting language instead.
Eventually it motivated me to switch the language for shell scripting for the second time, and nowadays I recommend Haskell's Turtle (https://hackage.haskell.org/package/turtle-1.5.16/docs/Turtl...) to anyone who is interested in safer shell scripting and still likes a concise and terse syntax (and it's blazingly fast too).
An application should never rely on a shell script. If you have to execute another program, use the syscalls (fork and exec on posix).
Could you explain why? I'm very happy with programs calling shell commands/scripts. You execute another program the same way whether interactively from a shell, or by calling system() from another program. The simplicity and universality of the call syntax is an advantage.
The problem is that very few people know shell scripting well, but once you get to know it, it’s not that bad, in quite specific cases.
Always use Shellcheck, though.
https://www.oilshell.org/blog/2018/01/28.html#shouldnt-scrip...
IMO, The inflection point where you should stop using shell is very low, though. The Google style guide for shell recommends that you should rewrite scripts once they are more than 100 lines, and I think that it probably too generous.
I wanted to create a custom git command and started with Rust/libgit2, but found it was missing too much. Shell scripting proved to be the most natural, best supported approach (though it was nevertheless very painful).
I think the main issue is: just because someone gets something to work on their system with their Python install, they think it works everywhere. And that is not at all true. There's a reason bin/sh|bash are so pervasive in administration, they are small, simple, well-understood, and resistant to the kind of bloat the Python chokes on.
python is basically this:
https://www.amazon.com/Wenger-16999-Swiss-Knife-Giant/dp/B00...