idank

You've hit the two top things on my to do list.. command substitution and control flows. It says on the top of the home page that they're not implemented yet.

Thanks!

I work full-time right now, so money isn't really an issue. I'll gladly accept commits though! ;)

Yes, you can parse without executing. I think shell is easier to parse than it seems, there are definitely some nuances that might be tricky to emulate, but I guess it's a matter of effort and how far you're willing to go. For syntax highlighting in IDEs, you probably don't need much, and maybe even a few regexs can get you close enough.

I actually spent a considerable amount of time going over bash's parser [0], and also zsh's [1]. Bash uses lex/yacc with some custom code. zsh seems to have a custom written lexer/parser.

Unfortunately none of them were written to be used as a library, as evident by global state, and a lot of hooks for prompts while parsing, etc. It might be possible to somehow make it work, but it seems like a lot of work to me. You might have an easier time looking at the parser I wrote, and extending that, or rewriting it in another language (as it's fairly small compared to the ones you find in shells). There's also libbash [2], haven't looked closely though.

[0]: http://git.savannah.gnu.org/cgit/bash.git/tree/parse.y

[1]: https://github.com/zsh-users/zsh/blob/master/Src/parse.c

[2]: http://www.gentoo.org/proj/en/libbash/

True, it doesn't figure out that some commands should be matched recursively since their arguments are other commands, xargs is also in the same boat. There might be an open bug for this already.

I only scanned sections 1 and 8 of the man page archive, so that's why files are missing.. but they can be added.

I'm tempted to add a place holder for all the variations of this forkbomb, until I teach the parser about functions.

You'll be amazed how many people try this (or other 'malicious' commands), presumably thinking I'm executing the queries. ;)

I think it'll be hard to match portions of the command to their boxes if there were no lines at all. How would you know that the `v` in `tar xzvf foo` matches the 4th box?

Having a single line when you hover could help, but sounds confusing for new users that are unaware of this feature.

I had something similar in mind. But this format means that repeated arguments that aren't adjacent will have their help text repeated.

Feel free to open a bug and we can discuss the options there.

It's definitely doable to add dedicated parsers for options of awk, sed, et al. But it's a lot of work... I doubt someone will go through the trouble of writing a parser for awk, just for this..

What is possible and quite easy is creating links for things that are explained in other tools, such as linking the regex argument of grep to some site that explains regular expressions, or somehow integrate it into the existing UI.

I can scan the access logs for the top explained commands, and show those on the home page (although that will probably only further increase their popularity ;).

Thanks for the incentive, but it's not needed and if I do get around to it, it'll of course be open source.

The main difficulty I had with writing a command line utility is figuring out the UI in a console. Suggestions are welcome of course.

Also, there are two possibilities for a client: the first queries an API on explainshell.com by sending it a command line. This has the advantage of the client being thin, and using the centralized man page database which is probably more accurate. But this means that you're potentially not explaining the exact command you're running locally, which may be confusing.

The second option is to run the man page parser, matcher, etc. locally against the man page on your machine.

Yes, I had this suggested on numerous occasions and gave it a go a while ago and it's indeed not trivial.

If a JS wiz would like to hack on this I'll gladly help and accept it.

I agree it's a little counter-intuitive. But if it's top to bottom then all the lines would cross each other which would ruin the UI. If there's a way to make it top to bottom while keeping the UI useful, I'd love to hear!

Thanks, that's really fun to hear.

ACK on the command substitution not working, with the current lexer that I have in place fixing this isn't easy and might take a while. But it's definitely up there on my todos.

Yeah, long pipelines are somewhat of a problem. You can currently navigate the commands with the buttons at the top, maybe they're not visible enough.

But I like the idea of being able to pin a particular line by pressing it, but I fear it might be a bit subtle and easy to miss.

Hi! Happy to see explainshell on HN again, thanks for all the compliments!

I agree about large commands being hard to follow. I tried solving this by adding the ability to navigate between commands. Right now clicking the command takes you to a page that displays that command options, but I can change it to the equivalent of navigating to it with the arrows at the top. There might be something better to do UI wise, though.

And what do you do if the dpkg'd version is a year older than the one you'd like to use and is available in pypi? Install just that one via pip? But what if that package has dependencies newer than what's installable with apt?

In the end you find yourself installing as much as possible with pip, and resort to apt for the really heavy stuff like numpy.

Does join scan the entire input to know the exact size and allocate that at the start? Doesn't it do the usual thing for resizing and that is to double the capacity? A generator will save a temporary list, it will however result in a lot of overhead for the generator switches. It might be beneficial if the input is really big.

Code is now on github: https://github.com/idank/explainshell

Which open source license would you recommend I use?

This regex is the one I used to extract options from man pages, not to parse the command line (which is simply a wrapper around Python's shlex).

It's possible, but I'm not sure how robust it will be. There are a lot of heuristics being used to guess what the options are, if they expect an argument, etc. So when feeding it with a command that was typed by someone, which is most likely correct, it's fairly easy to look up each option in all the possibilities. Not to mention some commands have a very complicated set of rules (e.g. find), and the position of some options is context sensitive, which explainshell isn't (it could be, but that would require a lot of manual work).

It's interesting because I had an idea once to do the opposite: see if explainshell could use completion files from bash/zsh/fish as another source of options besides the man page.

The regex is already getting out of hand ;) https://gist.github.com/idank/6382427

At some point I wanted to convert it to a DFA because I thought it will give me more control and flexibility and be more maintainable over time. The thing to look out for is not make it too permissive so it starts catching things that aren't really options.

Ha, cute! I'll add an API soon enough to return some JSON, so command line tools could wrap it more conveniently.

Thanks for reviewing it.

I definitely went into this with a 'let's start with something good and useful for as many commands as possible', so I haven't focused on accuracy for those edge cases. But from what I played with it so far, results have been pretty good.

Agree again on the pipe et al. support. It's not easy, hence why I haven't added this right out of the gate. I basically have to build a parser for bash, although I can probably get away with handling only a small subset of it that is interesting in this context, such as pipes, redirections, etc. I still have to figure out how to display multiple commands, because chunking them all in one page isn't going work visually.

I'll be happy to hear more about turning this into a learning environment. It might not work for this particular tool, but maybe it can use the foundations.

Yes they are, and most of the time require manual tagging. I fixed 'tr', try again ;).

An argument to an option without a space does work. When they don't it's because the option was incorrectly analyzed as having no arg.

The order of help blocks is determined by their position in the command and if they're on the left or right side of the lines.

If you always keep the order, lines would have to cross each other which looks less pretty and pretty confusing.

I don't have a Mac but I hope once the code is open someone will jump on it.

Re other flavors: do you know if there are easy ways to get a dump of all man pages, similar to Ubuntu's man page archive?

Will do, thanks!

I actually experimented with that but getting the lines to display correctly when scrolling was tricky.

Hi,

I'll open source the code sometime in the future so github would be the place for bug reports.

As for long commands, yeah they're a bit crazy right now since they get wrapped. I'm still trying to figure out the best way to display it.