Hacker News Reader

Comment by nikisweeting | Hacker News Reader

I consider shellcheck absolutely essential if you're writing even a single line of Bash. I also start all my scripts with this "unofficial bash strict mode" and DIR= shortcut:

    #!/usr/bin/env bash
    
    ### Bash Environment Setup
    # http://redsymbol.net/articles/unofficial-bash-strict-mode/
    # https://www.gnu.org/software/bash/manual/html_node/The-Set-Builtin.html
    # set -o xtrace
    set -o errexit
    set -o errtrace
    set -o nounset
    set -o pipefail
    IFS=$'\n'

    DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

I have more tips/tricks here: https://github.com/pirate/bash-utils/blob/master/util/base.s...

View on HN

I've had a thought percolating (for many months?) but I haven't tried to phrase it and I'm leery it may sound condescending (this is also more of a public address than direct response) ...

Shellcheck tends to steer us away from "weird" parts of Bash/shell and will happily suggest changing code that was correct as written. There are good reasons for this (and I still think most scripts/projects should use Shellcheck), but Bash is a weird language, and there's a lot of power/potential in some of the weird stuff Shellcheck walls off.

If I had to nail down some heuristics for working with it...:

- do lean heavily on Shellcheck for writing new code if you're unfamiliar with the language or are only writing it under duress

- don't implement its suggestions in bulk (the time you save on iterating can easily be blown later trying to debug subtle behavior changes)

- don't apply them to code you don't understand (if you have the time and interest to understand it but find yourself stuck on some un-searchable syntax, explainshell may help)

- don't adopt them without careful testing

- do take Shellcheck warnings about code that is correct-as-written as a hint to leave a comment explaining what's going on

- do leave Shellcheck off (or only use it intermittently) if you're trying to explore, play, or otherwise learn the language

That advice is applicable to any kind of code analysis tool.

Fair :) but I think there's a degree difference here (and this is more about the language than Shellcheck itself).

Also, maybe my original phrasing left room for misinterpretation? Shellcheck will happily suggest changes that break working code. I think I've had it happen ~4 times this year?

Obviously it varies a little by tool type and ecosystem--it's obviously not much of a surprise if a dead-code analysis tool suggests removing things you can't actually cut, and likewise linters in most languages can point out unused variables/parameters that you can't cut. But I can't recall the last time another tool gave me specific do-x-not-y suggestions that just weren't anywhere close to fungible.

> Shellcheck will happily suggest changes that break working code.

Which is OK once it is known. Your example nicely shows that. The flagged behavior is often unexpected, so there where it is expected having shellcheck line that turns off the warning greatly improves readability of the whole script, essentially showing the reader "here is the dependence on non obvious behavior."

Where such dependence exists, I prefer to see the "shellcheck" line before pointing to it (by turning that specific warning off exactly in that line). It's then a "vetted" and "explained" line.

For me this is way too much to invest in Bash. Any non-trivial utilities should be written in a real programming language when possible, especially when they are used in a context where they are likely to accrete complexity over time. I’m sure there are good use cases for complex Bash programs, but we should be reaching for something with less footguns by default.

(Shellcheck is an amazing utility and I use it for all Bash that I write.)

What is? Did I suggest anyone write complex Bash programs?

Shellcheck is amazing. As I suggested in the previous post, I use it for most of my own shell scripts/projects, especially stuff I intend to release. But, because shell and Bash are weird and full of pitfalls, I think it's worth making sure people know you can't just Shellcheck-and-ship.

I think his nitpick was you promoting weird bash tricks. They're neat to use but usually a pain for someone else to maintain, as a general rule, don't use them, don't try to be clever unless you're 100% sure that it's code you'll be the only one to see (which can rarely be guaranteed).

What would you suggest for emdedded systems?

lua, all day every day. It’s well understood, slim and plays very nicely with C which is how most embedded dev is done.

> and there's a lot of power/potential in some of the weird stuff Shellcheck walls off.

Can you give an example? Something Shellcheck warns you away from but which, if you used it, would be better (more expressive / more powerful / more whatever) than a shellcheck-approved solution?

Intentional word-splitting is a common one. So, for example:

    extract_nix_version(){
      echo "$3"
    }
    
    get_nix_version(){
      # shellcheck disable=SC2046
      extract_nix_version $(nix --version)
    }
    
    $ nix --version
    nix (Nix) 2.3.4
    
    $ get_nix_version
    2.3.4

This gets the information without another external utility (grep, cut, sed, awk...)

It's well-past bedtime, so for now I'm just leaving the first one I can think of (it's top-of-mind since I Shellchecked it in the last couple weeks... :])

It's a little contrived in this case, but another common example is a related suggestion for `read -r`:

    get_nix_version_parts(){
      local major minor patch
      # shellcheck disable=SC2034,SC2162
      IFS="." read major minor patch < <(get_nix_version)
      local -p
    }

    $ get_nix_version_parts
    major=2
    minor=3
    patch=4

Using -r there works fine for me, though your SC2034 complaint is valid, in that it essentially prevents from you from using a very specific built in formatter provided by "local -p".

This feels like a rare case, and I'd have no problem doing the "disable=SC2034" there, but I'd add a note explaining why.

Even then, one could make an argument (I think I would, actually) that an explicit printf is clearer and more robust. For example, it gives you more control over the formatting, and it doesn't introduce implicit coupling between the output of "local -p" and our function, which in theory could change in the future or be different on different bash versions, etc. I suspect this probability is very low in the particular case of local -p, but still it's not a great practice. Not something I want to give the stamp of approval of inside my code base.

    get_nix_version_parts(){
      local major minor patch
      IFS="." read -r major minor patch < <(nixv)
      printf '%s=%s\n' major "$major" minor "$minor" patch "$patch"
    }

Good catch on -r not mattering, here. I think `read` is one I've had trouble with and assumed that was it since I bothered disabling it. But after grepping around a bit more, the only one I see that shellcheck would object to that actually differs isn't common enough that I'd gripe about it.

I don't actually have a gripe with 2034, here. It's a good example of a statement that the user obviously has to triage.

Likewise, I'm just using `local -p` as a cheap way to show that the variables populate, it wasn't part of the original code I adapted. Yes--the case for coupling to the output of local -p is when the output is going to be used to set variables again later.

Here is shellcheck compliant solution which uses no external utilities, and is explicit about our intention:

    # function to imitate "nix --version" since I don't have it installed
    nixv() {
      echo 'nix (Nix) 2.3.4'
    }

    read -ra arr < <(nixv)

    echo "${arr[2]}"

  >    get_nix_version(){
  >      # shellcheck disable=SC2046
  >      extract_nix_version $(nix --version)
  >    }

You might do something like:

  get_nix_version(){
      local IFS=' '
      extract_nix_version $(nix --version)
  }

if you want set "IFS=$'\n'" globally, and still be able split on space "locally".

Edit: Still needs the "# shellcheck disable=SC2046" stanza, though

Thanks for asking this, by the way.

Going back to intentionally look through a few things for examples is helping me better clarify why I've had this impression slowly building up...

1. Some Shellcheck code titles/short messages communicate uncertainty well, but some others can read as more confident/absolute than the situation merits.

2. Broadly, Shellcheck tends to steer people away from word-splitting. Word-splitting can be a surprising PITA, so I get it. But, it's also a critical concept for understanding shell.

If you are reluctantly writing Bash/shell, it's good to have Shellcheck help you avoid it. If you want or need to understand the language, you're going to have to grapple with word-splitting to get there.

I have created a similar library for bash. However, there are more pitfalls related to errexit, see [1] and even shellcheck can not help there. I have tried to solve the pitfalls in my library, but it turned out to be ugly and unreliable. That's why I'm trying to use rust [2] for shell scripting like tasks nowadays.

[1] https://github.com/anordal/shellharden/blob/master/how_to_do...

[2] https://github.com/rust-shell-script/rust_cmd_lib

Your [1] doesn’t mention it, but I find shopt -s inherit_errexit to be indispensable.

https://twitter.com/hellsmaddy/status/1273744824835796993?s=...

You've a lot of cool stuff in that repo! I hope you don't mind me 'stealing' some stuff for a kind of booklet I'm working on with several shell tips and tricks.

https://github.com/kidd/scripting-field-guide/

PS: Any comments/PRs are super welcome there too

No problem, it's all MIT licensed so it's not stealing haha. See my parent comment with my full reading list as well.

If you want to link back to my repo as a cited source/credit that would be nice, but is not required :)

Cool thanks! I will link back and credit you :)

Note that this same wiki has a good page on the numerous surprising behaviors of errexit: https://mywiki.wooledge.org/BashFAQ/105

My perception is that there is no consensus on whether or not to use it. I go back and forth, but I probably use it more often than not.

imho all the behaviors that post outlines as "confusing" are my desired behaviors most of the time. For me it's pretty clear-cut, error out by default and only continue when explicitly allowed with something like `|| true`.

Here's my full bash reading list:

- ShellCheck: life-changing BASH linter and testing toolkit https://github.com/koalaman/shellcheck

- 30 interesting commands for the Linux shell – Víctor López Ferrando https://www.lopezferrando.com/30-interesting-shell-commands/

- 7 Surprising Bash Variables https://zwischenzugs.com/2019/05/11/seven-surprising-bash-va...

- anordal/shellharden https://github.com/anordal/shellharden/blob/master/how_to_do...

- barryclark/bashstrap https://github.com/barryclark/bashstrap

- BashPitfalls : Greg's Wiki http://mywiki.wooledge.org/BashPitfalls

- Common shell script mistakes http://www.pixelbeat.org/programming/shell_script_mistakes.h...

- Comparison of all the UNIX shells http://hyperpolyglot.org/unix-shells

- Defensive Bash Programming https://www.kfirlavi.com/blog/2012/11/14/defensive-bash-prog... or https://jonlabelle.com/snippets/view/markdown/defensive-bash...

- Bash FAQ and Cookbook https://mywiki.wooledge.org/BashFAQ

- Detecting the use of "curl | bash" server side https://idontplaydarts.com/2016/04/detecting-curl-pipe-bash-...

- Gensokyo Blog - Use Bash Builtins shell,fish,bash https://blog.gensokyo.io/a/fafbe742.html

- Rich’s sh (POSIX shell) tricks http://www.etalabs.net/sh_tricks.html

- Shell Scripts Matter https://dev.to/thiht/shell-scripts-matter

- Shell Style Guide https://google.github.io/styleguide/shell.xml

- Shellcode Injection - Dhaval Kapil https://dhavalkapil.com/blogs/Shellcode-Injection/

- Something you didn't know about functions in bash http://catonmat.net/blog/bash-functions

- Ten More Things I Wish I’d Known About bash https://zwischenzugs.com/2018/01/21/ten-more-things-i-wish-i...

- Ten Things I Wish I’d Known About bash https://zwischenzugs.com/2018/01/06/ten-things-i-wish-id-kno...

- Testing Bash scripts with BATS https://opensource.com/article/19/2/testing-bash-bats

- Testing Bash scripts with Critic.sh https://github.com/Checksum/critic.sh

- Useful BASH and UNIX commands https://cb.vu/unixtoolbox.xhtml

- When Bash Scripts Bite :: Jane Street Tech Blogs https://blogs.janestreet.com/when-bash-scripts-bite/

- Bashible: Ansible-like framework for bash-based devops https://github.com/mig1984/bashible

- Auto-parse help text from comment at the top of script https://samizdat.dev/help-message-for-shell-scripts/

- Make bash scripts safer by writing them in Rust https://github.com/rust-shell-script/rust_cmd_lib

- Additional shell options for non-trivial bash scripts https://saveriomiroddi.github.io/Additional-shell-options-fo...

set -euxo pipefail

Every script. Good enough.

Ahh, errexit:

  unless
       the command that fails is part of an until or  while loop, part of an
       if statement, part of a && or || list, or if the command's return status
       is being inverted using !.

All hope is lost.

Same, but I also do "shopt -s dotglob" so that * includes hidden files.

That's really fun to pronounce.

ooh good one, I'll add that to my snippet

errtrace is new to me, otherwise I do similar things.

set -ueEo pipefail

"pipefail" is also available in the FreeBSD Almquist shell, but not in the Debian Almquist shell.

Reply on news.ycombinator.com