UP: A tool for writing Linux pipes with instant live preview | Hacker News Reader

UP: A tool for writing Linux pipes with instant live preview

github.com

485 pointsducktective83 comments

For fast iteration with pipes, using only basic unix tools, and arguably less risky to fat-fingering, I write it in a file, and use watch -n1. Easy to modify, temporarily comment parts of it, etc...

Also, I wrote this plog function (for pipe-log, or something):

    plog() {
      local msg=${1:-plog}
      tee >(sed -e "s/^/[$msg] /" | cat 1>&2 )
    }

That allows me to see what's happening in the middle steps of the pipeline.

I ended up compiling a list of tricks like that in https://raimonster.com/scripting-field-guide/ , in case anyone is interested (and feedback is appreciated)

rgrau, I'm halfway through your field guide. I appreciate how concise the text portions are, and how, although each section teaches one thing at a time, you are sometimes willing to leave the reader with "here are all the major incantations, figure out the details of each one and appreciate them" - such as in section 4.4.

Learned several new bashisms which i will put into my computer tip memorization deck. Thank you for writing.

Thanks for giving it a read. Glad it was useful!

Yes, I also found it very good. Opened a PR[1] to "Awesome Shell" to include it.

[1]: https://github.com/alebcay/awesome-shell/pull/424/files

You can skip cat.

  tee >(sed -e "s/^/[$msg] /" 1>&2)

This is super cool. There's so much good stuff in there I'm gonna have to come back a few times just to digest it all! Also looking forward to some of the links at the bottom.

Do you have a standard template you use for your scripts? You kind of touch on this in section 4.20.1 but don't go too deep.

Also have you done any work with writing zsh (or bash) completions? I feel like having good completions would make my scripts magnitudes more handy, but I've had trouble finding good starter resources.

Most of my scripts start with that template on 4.20.1. That's the baseline. But soon enough they have 4.22 and 4.25.

I never liked those "bare minimum templates" that are 100 lines, because usually the scripts I write start being just a couple dozen lines. If they grow, I add things on a need basis, but the ratio of logic/template has to be >0 always.

wrt completions, doing smart completions is hard, and the zsh ones are not very intuitive to me. I try to make my scripts work with "compdef _gnu_generic ". It goes a long way, and it's an incentive to write proper --help

If you don't mind, this is my base template for Bash scripts:

https://pastebin.com/FYMmnZ0B

At the start it provides a stacktrace function which can also be used to notify you if your script has failed (e.g. via xmpp, e-mail, etc.). It is not my own creation, as I found it somewhere a few years ago, but found it super useful over time.

In addition, there are two versions for checking for dependencies. The first lets you communicate instructions for every tool that is required while the second can be used if you simply want to communicate the list of programs that is missing.

The third part is about named parameters, which can be helpful if you don't want to use positional parameters.

Very reasable, I like the balance berween just writing code and the commenting in how it works.

I wish somebidy had done this for python3.

I use a similar tool in Emacs called piper https://gitlab.com/howardabrams/emacs-piper

And here's the EmacsConf video for those interested: https://www.youtube.com/watch?v=HKJMDJ4i-XI

Piper looks amazing, shame its not on a package repo.

I didn't manage to grasp it from quickly skimming the video and the gifs: is it possible to retry a command in piper until I find the right combination of flags, and only then press `|` to push it to the buffer? Ideally, keeping the incomplete command visible for editing all the time? Also, can I get back to older contents of the buffer, i.e. a couple `|`s back? If yes, that might be basically what I wanted up to be from the beginning.

> WARNING: Please be careful when using it! It could be dangerous. In particular, writing "rm" or "dd" into it could be like running around with a chainsaw. But you'd be careful writing "rm" anywhere in Linux anyway, no?

Side effects always seems like potential disasters for me.

if linux had openbsd's pledge(2) then you could disable filesystem writes, which you don't need for grepping. I know linux probably has orher mechanisms to achieve the same result but in this situation pledge seems natural.

Probably that’s right. But note that sort, for example, may need to write to (temporary) files. So disabling writes entirely is probably not a sufficient solution

You can achieve the same thing in Linux with seccomp.

Why side effects? It's about the main effect of tools like rm or dd. Some things just don't fit to live previews like provided by UP.

You don't want to accidentally delete or write to things and this command seems like an accident waiting to happen. A command whitelist would be very helpful but it doesn't look like UP has one.

Yes, even a blacklist with obvious ones listed would be cool. Imagine just typing the characters rm and you're gone..

Good point. That's a big problem. I could conceivably have a perfectly harmless tool where a prefix of the name is something dangerous.

One of the reasons for not doing this is that I decided I don't want to lure the users into a false sense of security. A comment by hnlmorg basically sums this up better than I could: https://news.ycombinator.com/item?id=26645875 (The other being, that I was cutting corners on my initial ideas brutally trying to push the tool out of the door, and it's still kind of a miracle I eventually managed to do that.)

I remember reading about a project that let you "preview" the effects (e.g. file changes) of terminal commands by using ptrace to intercept the syscalls and just logging them instead. Maybe you could do something similar here to prevent accidental file deletions.

My first thought was to sandbox the executable but this sounds even better. Is it hard to implement the interception of all syscalls? And can file reads be distinguished from file writes?

Not terribly hard to do, and yes you can distinguish reads, writes, etc. A tool to log syscalls to see what might be possible is strace, it's incredibly useful for debugging a misbehaving application that either you don't have all the source for or it's gigantic codebase that you don't understand. Seeing which files a program is looking for is often all you need to figure out why it's behaving the way it is (i.e. locating an undocumented/badly documented/incorrectly documented config file)

Wow that would be cool

That's maybe: https://github.com/p-e-w/maybe

Please note that up respects $SHELL, so if you find a way to set $SHELL to point to maybe (possibly wrapped in some helper script) in a useful manner, it might actually work! I'd highly appreciate an issue with a report/guide for that then on up, others might find that helpful!

For details, see: https://github.com/akavel/up/releases/tag/v0.3

I'm not affiliated with maybe, and it's also no longer maintained. I don't know if there's a newer, better tool out there.

Chrome's developer tools have live preview, and they apparently accomplish this at the VM level by blacklisting a set of impure bytecodes when running the code in a separate evaluation context. If one of the bytecodes is found then it throws an exception.

I kind of wish LuaJIT had something like that.

Is there a way to start a new instance of bash with read-only permissions only? This could easily fix the side-effects issue.

The developer, Mateusz, did a "show HN" some 2.5 years ago:

https://news.ycombinator.com/item?id=18291302

Hi ducktective, thanks for submitting again! :) It was a pleasant surprise seeing it here today, on my regular HN addiction^H^H^H daily read :) I completely didn't connect the dots that it's ok to repost periodically to share the fun and serve as a refresher for the lucky 10k, kudos to you for being here to do that :)

I kinda hunt for cool UNIX tools periodically :D Someone mentioned it in a recent post on fzf, I checked out the repo and posted here.

Took me a while to get the lucky 10k reference ;)

https://xkcd.com/1053/

'twas me ;P and the timing coincidence was suspicious indeed xD

HN should have a separate section and do the annual re-post of the most cool stuff =)

This looks quite good, and I’ve wanted something like this for a while now. I wonder if it could be improved by incremental versions of commands like grep, e.g. making the change grep net -> grep network faster.

Another system with some different but related capabilities is userland: https://hisham.hm/userland/

That is some impressive Proof of Concept and it's not even surprising considering who created it. Using a "terminal" like that would probably change a lot of how we work on the command-line

Wow this is almost exactly what I’ve been looking for for years. I don’t want a tool like up that works on a fixed entry command; I want a full terminal that takes care of everything, showing me the output at every step. What’s missing is some caching (think HTTP requests) and limitation (e.g. evaluate the first 4 HTTP calls, and wait for me to approve the rest if they’re good)

A variation on userland is the next project I'm trying to work on, though there are... distractions. In a random case that some benevolent millionaire wants to sponsor me to work full time on things like that, I'm certainly open to being contacted, my email is in the commits ;)

As to another huge and absolutely awesome emerging contender in this space, see https://luna-lang.org

Now, that one was a futuristic tool! Thanks for sharing! (He also wrote htop :D)

https://github.com/hishamhm

I started a very similar side project (in Rust) at the beginning of lockdown, but didn't manage to keep up as I had to spend most of my "free" time parenting.

Although the self-proud part of me feels a bit disappointed that someone else got there first, I'm actually glad that it happened (it also means that my idea was sound). I didn't even realise that something like that existed already. Kudos to both the developer and the OP!

Just saying, the tool is about 3 years old, most recent commit is 12 months ago..

I noticed, that's why I wrote "I didn't even realise that something like that existed already" :) And I'm grateful to the OP who brought this tool to HN's attention.

Actually, 5 months ago :)

Don't worry, when I published up I also learned about a previous similar project from commenters ;) probably wouldn't start if I knew earlier ;) fortunately I didn't know :D just in case if you still have some mojo left, up is very open to an idea of a RIIR contender, it's actually even kinda encouraged in the readme :)

Slightly related, I wrote a tool called ptail[1] that only shows the last N lines of output from a pipe. Quite useful in some situations

1. https://github.com/orf/ptail

I didn't realise I needed this until I saw this post. Now I'm wondering why this isn't the standard already for every command line out there. Seems like a no brainer improvment with non to few downsides.

I've been working on a way to implement this into my own shell (murex). The shell already has a builtin support for known "safe" commands, that is commands that are generally safe for background execution. The reason murex has this feature is because it supports executing the pipeline to generate more relevant autocompletion suggestions (eg when picking out nodes from a JSON or YAML file). Thus implementing an `up` like feature should be a pretty easy addition....however I ran into a bug that stalled the development of said feature and it's been in that stalled state for the last ~2 years.

The reason development stalled for so long is because personally I'm not sure this is something that should be widely supported. Even with murex, which has an internal catalogue of known safe(ish) commands, the dangers of tools like this is pretty high. Running this against a regular shell which doesn't even do that seems like potential suicide.

How about spinning up a container with the entire host file-system mounted in a Copy-on-Write configuration. This way you can trash the container as much as you want without affecting anything on the outside.

Combine it with syscall logging and you might actually have a great tool for figuring out exactly how a command is affecting your system.

That'll certainly help but you'd still have issues with any connected devices (USB, network API alls, etc).

I don't think it is a solvable problem though. There is no way to know ahead of time which usage is safe and which is destructive because of the variety of UNIX-like configurations out there, variety of coreutils and variety of optional flags. Not to mention all the 3rd party tools out there. And some seemingly destructive usages are vital for safe usage (eg tmp files). Worse still, by the time you build a safe container you've have spent more time spinning up a sandbox than you've had spent using the CLI preview tools anyways (yeah ZFS could help here but few people run ZFS on their dev machines and you still haven't solved mocking APIs for connected devices).

I just can't see how this can be achieved safely without completely redesigning how SYSCALLs work from the ground up to follow a method more akin to iOS or FireFox's on-demand permissions. At which point you've gone well beyond the realm of "yak shaving"

Network API calls seems like a scary one. But for things like USB and sys-calls, I'd suggest a "block and log" method.

Like "running this command shows this output (with some extra error messages" and tried to delete these files and do these system calls.

But my point is even the safest of coreutils will make dozens of SYSCALLs. The moment you start pipelining commands there will be several writes since that's how pipelining works and if you break even one of those writes the entire pipeline fails and the usefulness of this pipeline preview utility falls flat.

There isn't a way you can make this utility safe and useful.

You can make it safe, but then you're drifting into a whole new field of computing with regards to analysing suspicious binaries. Unfortunately you then don't have reliable, accurate, real time pipeline previews. Or you can have the real time previews but you then have to accept there is some risk involved. But you can't have it both safe and risk free.

This is why murex took the approach of having a safe list of trusted executables. It doesn't remove the risk but it at least reduces the risk to a subset of commands that are typically read only. However even that is far from a perfect solution.

Thinking about it more, the kind of guarantees you want are the kind given by a VM for running untrusted binaries. The only difference is that you don't mind read-access to the host system.

Yes, there will be certain things that involve system-calls or network activity that your previewer will not accurately show. But those cases fall far outside the goal of "live preview of text output of data processing pipelines".

No they don’t. Believe it or not a lot of people make network API calls from the command line.

I know this because I’m one of them. I’ve even published guides I’m on how to make said API calls and parse them all as one command pipeline.

Sounds like SELinux might do the trick here :)

How do you differentiate between a write which is safe and a write which isn't? File system location gives a hint but it's no guarantee. Likewise for network connections, SELinux couldn't protect against an unsafe HTTPS POST request vs side-effect free GET request.

And the amount of SELinux tuning you'd need to cover every single executable in every single configuration available on SELinux enabled machine (and even then, your limiting your tool to only work on systems with SELinux) is monumental, manual, inevitably error prone.

There isn't a practical way of making this safe.

This thread was originally about using a container / overlay FS that has read access to the underlying filesystem, and writes happen into a temporary overlay.

Network connections and certain system-calls might still be scary. However, VM / container systems that are supposed to defend against hostile hosted machines should also be safe here. The only difference is that here information leaking from the host into the VM is the point, instead of dangerous.

No, this thread was originally about a version of said tool that I built and the dilemma I had about making it safe.

A VM would be too slow to spin up to be practical for a live preview. A container would be too host specific. And neither solutions fully solve the problem because all you’re addressing is file system access and, as mentioned several times already, that’s only scratching the surface of the interactions a command line tool can make.

My shell, murex, is designed around making networking access as seamless as writing to a local file. Bash does this too with its /dev/$datagram/$port pseudo file system too. So you can’t just dismiss non-local changes as irrelevant.

You're right, it wouldn't be perfect. I was being a bit facetious :) A perfect solution is never really possible and in this case, if you can't have pretty good guarantees of getting it right, probably best not to try. The solution here might just be to be very careful when using such a tool.

There was this same idea realized via fzf preview, also posted to hn I think.

fzf has been submitted many times - https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... - but most recently:

Improving Shell Workflows with Fzf (jambor.dev)

https://news.ycombinator.com/item?id=26634419

221 points by todsacerdoti 22 hours ago | 51 comments

Maybe I'm just old and grumpy, but what do you gain by this compared to just building up your pipe directly in the shell? Not having to hit enter once in a while?

You still have to press Enter.

I'm with you. I can't see myself preferring this over using my shell's interactive prompt, history, and a use of head/less at the end.

Thanks to the author and OP for sharing and it's good for other people if they find this useful, though.

EDIT: Maybe I could find this useful when the command that provides `up`'s input is not something that can be repeated for the same input. But even then, you can put it in the background with its output redirected to a file, then make the command that you would build inside `up` from a pipe like `tail -fn+1 bg-job.out | cmd`.

Not much, really. Yet somehow those little things add up to something... faster :) Basically "just":

- not having to append `| head -n $ROWS` or something similar on every command (yet having the view as-if-you-typed-it, but at the same time being able to scroll down further through the results; and having them visible as you continue tweaking, unlike `less` which you have to quit to tweak the pipeline),

- not having to press up-arrow-key, then alt-b / alt-f to a position in the pipeline you want to tweak after every Enter (e.g. just before the `| head...`, but sometimes a bit more back),

- sometimes also not having to dump the initial input from a command into a file, and to wait until it's finished, but being able to immediately start filtering even while the input is still being generated.

Once I just had those ideas, I felt pain every time I repeated the dance, enough that it made me push on up even when I was super tired of working on it. The pain of not having it when I know it could exist was even bigger.

Somewhat similar is fzz https://github.com/mrnugget/fzz

Feels like a perfect tool for taco-bell style programming(googleable), and i think there needs to be more in this style.

Powershell for example has an IDE(MS style) that is damn helpful by providing breakpoints/intraspection but this sits at a lower/faster level for just inspecting the initial pipings.

Powershell is probably the closest shell to the REPL experience of Xerox PARC and ETHZ workstations.

UNIX shells could approximate it with support for shared objects, D-Bus/XPC/IPC, but few make use of it.

I'm interested to know how and in what ways the REPL experience in Xerox PARC and ETHZ workstations were enjoyable...Can you elaborate please or share some links?

I know that there is Nushell (1) that tries to re-do the concept of shell with ideas taken from Powershell.

[1]: https://github.com/nushell/nushell

When whittling down text output like the examples on the project page, I've found that just piping into vim and using a combination of vim commands and external commands like `:%!grep -v foo` to be far easier and much more capable.

I thought I'd seen almost everything in bash, but I've never seen |& before. My attempts at googling it aren't coming up with anything, can anyone advise what it does compared to a normal pipe?

From https://www.gnu.org/software/bash/manual/html_node/Pipelines...:

> If ‘|&’ is used, command1’s standard error, in addition to its standard output, is connected to command2’s standard input through the pipe; it is shorthand for 2>&1 |. This implicit redirection of the standard error to the standard output is performed after any redirections specified by the command.

Ah ha, I suspected it was that but couldn't find the relevant docs. Thanks!

Just recently I thought I could use a live checking tool for regexes but without the horrible UI of the typical website made for this purpose.

I will definitely try this and see how it combines with ripgrep/rg.

regex101.com is pretty comprehensive, my go to for complex regexes.. not sure if that's what you're complaining about..

That's what I tried recently among others, it's the website equivalent of an IDE. And when I actually started entering some regex it started throwing errors and timeout messages for inputs that standard grep will happily chew through in fractions of a second.

On subpar hardware that site is a single collection of annoyances, including window-filling loading screens.

That said I tried UP and it works very nicely for what I need.

I was looking for a very similar tool for my project. Anyhow does some one know about a tool that allows you to render your commandline output to html

Depends on what markup language that output is in. If you just want a conversion of colors and ANSI escape codes to HTML tags, I've recently found and used this script (1). This SO link discusses few other options (2).

[1]: https://github.com/pixelb/scripts/blob/master/scripts/ansi2h...

[2]: https://unix.stackexchange.com/questions/44956/converting-co...

Thanks for sharing these and I think these really help

I find `fzf --bind "change:reload:some-shell-command {}" --disabled` a better alternative.

Wow I literally just thought about building a tool like this yesterday! Spooky.

This seems amazing. I can’t wait to try this.

very good idea. thanks akavel

“Running around with a chainsaw“

Lol!

Reply on news.ycombinator.com