Lessons learned from writing ShellCheck
vidarholen.net
vidarholen.net
O_o There must be more to it then that. What the hell does anyone do in their spare time at home for fun then?
One thing I've really appreciated is the Github wiki pages for individual shellcheck errors. Without those pages I wouldn't have learnt so much more about shell and the shellcheck tool would be more difficult for people to use :)
The vast majority of Software Engineers I’ve come across don’t write OSS code, don’t write blogs, maybe read Hacker News. For most of them this is sort of a non issue.
It might be a non-issue as in "I probably wouldn't have exercised that right anyway", but that doesn't mean that one feel the restriction is acceptable, fair, or justified.
I would not. Regardless of whether I had any intention to do it or not.
More realistically... most engineers don't code as a hobby. HN, /r/programming, and other software dev hangouts are small echo chambers.
This sounds unnecessarily dismissive to me and also unsubstantiated. It removes the focus from the more important fact that this is a foul encroachment of personal freedom regardless of how many professional coders like coding as a hobby.
Anecdotally, the programmers I work with don't seem to read anything outside of StackOverflow or contribute to any OSS projects. Albeit, what OSS projects we contribute to or what communities we participate in isn't usually a topic of conversation among my coworkers.
> It removes the focus from the more important fact that this is a foul encroachment
I agree that this is an encroachment on personal freedom and I doubt this clause is legally sound (in California at least). I am quite surprised that Apple has such a policy. But I am even more surprised that they are able to find programmers who agree to it.
You can't compete with Apple, which is fair. However, you're also not privy to what Apple is or isn't working on at any given time.
If Apple gave you permission to work on something, you could infer that they're not working on it, so it's not allowed.
The statement reads simply as: apple works on many things, and doesn't want you working on things outside of apple that may compete with current r&d or future r&d.
We can disagree with them on if it seems appropriate, but it doesn't seem to be about their software quality at all.
And yet Apple is totally fine with benefiting from the open source work of others...
And? Every other person benefits the same with the code contributed under the licenses in play. I fail to see the issue. With llvm for example they seem to be upstreaming a lot of their xcode backend stuff as they get time. So that statement of many copyleft proponents that only copyleft encourages upstreaming rings hollow to me.
Hell go talk to people from Redhat about custom gcc forks that target chips that aren't upstreamed to gcc. Just because you're using gcc and modifying it doesn't mean you'll actually be contributing the code if its all internal.
It seems to me that they are afraid (or at least unwilling) to compete on even grounds, despite likely having more budget than a single random employee competing in their free time.
Is there a lawyer with experience in the field reading this?
In summary. They are allowed to some extent, but if a court would consider a non competition paragraph as being too broad, the paragraph becomes void in its entirety, as if it was never entered into the contract at all.
For non management employees the exclusivity must be quite specific, and unless the law has changed recently without my knowledge, and entire industry, or a broadly specified skill would fail the test in any case I can think of.
Applying the most basic levels of those principles, I would assume that attempting to limit a programmers ability to engage in programming in general would lead to nothing but a complete dismissal of any claims related to the non-compete clause if someone brought it to court.
Effectively, if you try to prevent someone to apply their trades and/or talents to broadly, they could essentially moonlight for your worst competition without much of a risk as long as they don't convey what is clearly trade secrets as they are differently protected.
Suffice to say that companies with decent legal departments tend to write specific non competition clauses especially outside of the realm of management positions, as they get no value at all from too broad non competition clauses.
I've used ShellCheck to catch/fix numerous bugs on some mission critical financial systems (that depend on shell scripts!).
Added it to pipelines and as a mandatory pre-commit hook in git, so all teams started using it (whether they like it or not).
It even caught some places where a rm -rf / could happen because default parameter substitutions were not set ( Same as the Steam bug where it would remove your home directory, but on some big financial systems :) )
All in all - an amazing tool that saved me from a lot of grey hairs throughout the years.
It seems so "obvious" in hindsight but not many other tools are so good.
Also, tangentially, this is also so perfect:
https://github.com/koalaman/shellcheck/wiki/SC2154
and, unless somebody proves otherwise, makes me believe that with ShellCheck and using the described conventions a shell script could be better diagnosed for mistyped variables than a Python script can (namely at "compile time" and not during the run).
Vidar, your work significantly improved the life of people confronted with shell scripts. Thanks!
Can it? That's what I would like to know, how to know without executing the code with the argument 4 that there's an error in this Python code:
import sys;
def f( x ):
if x == 4:
x += q
print( x )
f( int( sys.argv[1] ) )
That will "work" until 4 is passed at the runtime: $ ex-undefined.py 4
Traceback (most recent call last):
File "/tmp/ex-undefined.py", line 8, in <module>
f( int( sys.argv[1] ) )
File "/tmp/ex-undefined.py", line 5, in f
x += q
NameError: name 'q' is not defined
With this shell code #!/bin/sh
f() {
x=$1
if [ "$x" = 4 ] ; then
x=$((x+q))
fi
echo $x
}
f "$1"
and shellcheck I get: Line 5:
x=$((x+q))
^-- SC2154: q is referenced but not assigned.
even if I have never executed the code. $ flake8 foo.py
foo.py:6:14: F821 undefined name 'q'
$ mypy --check-untyped-defs foo.py
foo.py:6: error: Name 'q' is not defined
(They also have some other complaints about your code that you'd either have to disable or adjust to.) Success: no issues found in 1 source file
and additionally produced a .mypy_cache folder of 2 MB at the place where the script was.So the style of "unreasonable" defaults of Python alone (not detecting the error, speaking from the point of view of a user of Perl) propagates to the "unreasonable" defaults of the checkers.
Still thanks Ded7xSEoPKYNsDd, I really wasn't aware of these! Yet, even if I haven't formally specified that, I was looking for the way to do it with the Python as the language and its default interpreter alone, as for Perl nothing additional has to be installed:
use 5.010;
use strict;
sub f {
my $x = shift;
$x += $q if ( $x == 4 );
say $x;
}
f( $ARGV[ 0 ] );
Gets me: Global symbol "$q" requires explicit package name (did you forget to declare "my $q"?) at ex-undefined.pl line 6.
Execution of ex-undefined.pl aborted due to compilation errors.
I am aware that for shell I'd need an additional shellcheck but Python is many, many times bigger than the shell binary alone (or even the sum of the shell and shellcheck binaries), and actively changed, whereas the shell semantics is standardized and effectively frozen in time, and from the shell interpreter alone a very low startup overhead is expected.I have heard from other Haskellers that you can sometimes get good performance by hand-rolling an application monad, and then writing out the MonadFoo instances so you get good ergonomics:
-- Instead of this:
newtype App a = App { runApp :: ReaderT AppEnv (ExceptT AppError IO) a }
deriving (Functor, Applicative, Monad, MonadIO, MonadReader AppEnv, MonadError AppError)
-- Try this:
newtype App a = App { runApp :: AppEnv -> IO (Either AppError a) } deriving Functor
instance Applicative App where ...
instance Monad App where ...
instance MonadIO App where ...
instance MonadReader AppEnv App where ...
instance MonadError AppError App where ...https://downloads.haskell.org/~ghc/latest/docs/html/users_gu...
It also gives you the ability to “tag” sections of code for profiling, so if you suspect that the internals of some named function are giving you grief, you simply add a tag (some pragma comment thing I think) and get profiling information for it. It’s much quicker than having to refactor just to profile sections of code.
Haskell Performance Patters - Johan Tibell (2012) http://johantibell.com/files/haskell-performance-patterns.ht...
Performance (Haskell Wiki) https://wiki.haskell.org/Performance
Detecting Space Leaks - Neil Mitchell (2015) http://neilmitchell.blogspot.com/2015/09/detecting-space-lea...
The Haskell performance checklist - Chris Done, et. al. (2017 - 2019) https://github.com/haskell-perf/checklist/blob/master/README...
ThreadScope (Haskell Wiki) https://wiki.haskell.org/ThreadScope
Top tips and tools for optimising Haskell - Will Sewell (2015) https://making.pusher.com/top-tips-and-tools-for-optimising-...
State of Haskell Ecosystem: https://github.com/Gabriel439/post-rfc/blob/master/sotu.md#c...
Guide to Haskell (2018): https://lexi-lambda.github.io/blog/2018/02/10/an-opinionated...
Haskell learner milestones: https://stackoverflow.com/a/1016986
GHC optimizations (rather advanced, but interesting): https://stackoverflow.com/questions/12653787/what-optimizati...
Compilation of various Haskell tutorials on more advanced topics: https://markkarpov.com/learn-haskell.html
Despite the popular saying, there’s a wealth of knowledge about Haskell beyond just academic papers, and in digestable form, but it’s scattered around little-known blogs. I’d recommend you to check out r/haskell from time to time, or subscribe to the Haskell newsletter - at least for me, those are the places where I got to know most of Haskell blogs.
"Parallel and Concurrent Programming in Haskell"
https://simonmar.github.io/pages/pcph.html
He also has some interesting articles like
https://simonmar.github.io/posts/2018-06-20-Finding-fixing-s...
I believe I know where some of the leaks may be coming from. I’m one of the authors of hadolint, the docker file linter. We used ShellCheck as a library to lint the bash code found inside the files.
When we attempted to compile Hadolint to Javascript to embed it on a web page, we discovered that the code in ShellCheck was not optimal. The culprit for our use case was the abuse of regular expressions for checking equality of bash snippets.
In Haskell, most regexp libraries interface with external C libraries and have pinned memory objects for performance, but one must be careful not to accumulate too many of them.
The solution for us to have a usable Javascript version was to reimplement the equality operator for those cases we needed, and now memory usage was down again.
We haven’t stumbled upon our first space leak yet.
What's a JSVM? I'm not getting anything from Google that I think makes sense in the context of post.