Comment by hnlmorg | Hacker News Reader

Comment by hnlmorg | Hacker News Reader

Parent Full thread

> Command line programs are black boxes (they could be pure bash, a compiled C++ binary, an npm package, etc), which means there is no programmatic standard for extracting the command set.

You say that but it's actually an easier problem to solve than it first appears. Both my shell, murex[0], and Fish[1] parse man pages to provide automatic autocompletions for flags.

I also have a secondary parser for going through `--help` (et al) output for programs that don't have a man page. This isn't something that is enabled by default (because of the risk of unintended side effects) but it does work very effectively.

They way I look at it though, is you want to cover as many things automatically as you can but there will always be instances where you might want to manually fine tune the completions. For that, I've also defined a vastly simplified autocompletion description format. It's basically a JSON schema that covers 99% use cases out of the box and supports shell code to be included to cover the last 1% of edge cases.

This means you can write autocompletions to cover common usecases and have the shell automatically fill in any gaps you might have missed.

[0] https://github.com/lmorg/murex

[1] https://fishshell.com

You should publish some docs about that! Maybe other tools can pick it up.

A couple years ago I wanted to tackle this problem for Oil, and there was some brainstorming here:

https://github.com/oilshell/oil/wiki/Shell-Autocompletion

https://github.com/oilshell/oil/wiki/Shellac-Protocol-Propos...

and on the oilshell Zulip in #shell-autocompletion.

The author of the Ion shell went pretty far with it, but it seems to be dormant now (or correct me if I'm wrong):

https://www.redox-os.org/news/rsoc-ion-ux-2/

https://www.redox-os.org/news/rsoc-ion-ux-3/

https://www.redox-os.org/news/rsoc-ion-ux-4-5/

To me a problem is that most shells like bash and even zsh are too impoverished to implement some kind of common protocol. For one, they don't work well with structured data.

I just looked at Fig, and it seems interesting as well. The approach of using Mac-specific APIs and doing it outside of the shell skirts that problem.

I did want to make Shellac correct and precise. Specifically it should have the invariant that if the tool suggests a completion, it should actually be a prefix to a valid command! And it was also supposed to separate the problem of completing the shell language from completing the argv language -- those are two different things. For example shells complete variables like $P -> $PATH, etc. And they complete their own builtins.

There's docs on the autocomplete schema but I'd wager that will be very much a NIH thing because it's an easy problem to solve but one that nobody seems to agree on. Thus everyone will look to solve it themselves. And I can't really complain about that because I did just the same myself -- rather than looking for an existing schema I decided to define one myself.

The docs on my schema is https://murex.rocks/docs/commands/autocomplete.html but it was a little rushed so could use more examples and a little more TLC.

Also this XKCD strip feels apt: https://xkcd.com/927/

OK thanks for the info. Well I'd say coming up with an approximate solution for one shell is easy -- but that's the state of the art. That's what bash, zsh, and fish do. (Not they are the same -- I'd like something closer to fish, not bash.)

The goal was to improve on the state of the art, which is hard. The original motivation for Shellac is here and I discussed it with many people. The Ion shell author wrote some code toward it, and so did a zsh dev.

https://github.com/oilshell/oil/wiki/Shellac-Protocol-Propos...

But it didn't really go anywhere for a variety of reasons. One is the "bootstrapping" problem. That is, there's no incentive for anyone to implement shellac if there's no corpus of completions. Likewise there's no incentive for the author of new-popular-tool to use it if no shells implement it.

The other problem is whether the completion is declarative or imperative, which you hinted at. If it's more imperative then you've automatically coupled it to a language like Murex. If not then I think the UI will be limited and it will break some of the invariants I wanted (but are maybe not strictly necessary).

I think that most people on the #shell-autocompletion Zulip wanted it to be declarative. I was one of the few people who said that declarative isn't enough, mainly because the ACTUAL command lines are written in imperative languages. It's sort of an argument around computational power.

To me the fact that all the popular systems are imperative is evidence in favor of that, until reality shows the opposite :)

I also wanted to push the burden of developing the logic on the people who write the tools -- THEY know what its syntax is. This also solves versioning problems. So the idea was that EITHER the shell or the TOOL ITSELF would be a Shellac server. In the latter case, it would never get out of sync, which does happen in bash, zsh, and fish.

(Although to be fair if you dynamically scrape --help, that also solves the versioning problem, but not if you bake it in. bash-completion actually DOES the dynamic execution and scraping of 'ls --help' on TAB, which I didn't know before implementing Oil's autocompletion )

One of the things that we discovered when trying to build out a declarative standard is that it can never encompass everything, precisely because many CLI tools implement their parsing logic imperatively! Our solution was to start off with a declarative schema, that can be supplemented with an imperative escape hatch.

Yes I'd like to see how the imperative escape hatches work for Fig and Murex (and how many completions needs to use it)

Actually I just remembered I enumerated some "unusual cases" here:

https://github.com/oilshell/oil/wiki/Diversity-in-Command-Li...

Is the escape hatch for Fig in JavaScript? One thing I would like to see "rescued" from Shellac is the idea of allowing the escape hatch to be in any language, not Oil or Murex, or JS. In particular, it could be the language that the tool itself is written in!

Reply on news.ycombinator.com