Hacker News Reader

Comment by svnpenn | Hacker News Reader

As another user mentioned, many POSIX and/or GNU utilities havent aged well. I respect trying to stay portable, but peoples needs change over time and these tools simply havent kept up. Like the other user, I use Fd now instead:

https://github.com/sharkdp/fd

As well as Silver Searcher:

https://github.com/ggreer/the_silver_searcher

while Grep performance is pretty good, its also gotten pretty stale with regard to its defaults and options.

View on HN

For grepping, rg is faster and saner.

https://github.com/BurntSushi/ripgrep

On end-user systems with fast I/O, it is usually a better use of resources to have a battery-/suspend-/reboot-aware indexing system that has path/extension white/blacklists and prioritizes file monitored changes. This way, searching can happen against an optimized text search DBMS much faster than waiting for zillions of IOPS that are basically wasted by instead spending them ahead-of-time in the background/idle indexing text files/metadata/structured data once when it changed/s and already know exactly where all occurrences of kWhateverCondition_PP3V42_G3H or \A[AB]{1,3}c+d\z live under ~/Projects without reading any actual files.

Do either of these tools have a "grep" mode, so that I can alias grep to it and get similar, but improved, behavior? 25 years of habit is a lot of reinforcement to overcome and I'd love to just be able to take advantage of these tools with an alias.

As is I just have a grep wrapping shell script to give it some sane defaults which works fairly well.

No, you're asking oranges to be bananas. Read the man page or user guide. It's not grep, it's better and already has saner defaults. https://github.com/BurntSushi/ripgrep/blob/master/GUIDE.md

The closest I know of is `rg -uuu`. From the manual:

> -u, --unrestricted ...

> Reduce the level of "smart" searching. A single -u won't respect .gitignore (etc.) files. Two -u flags will additionally search hidden files and directories. Three -u flags will additionally search binary files.

> rg -uuu is roughly equivalent to grep -r.

Author of ripgrep here.

> As is I just have a grep wrapping shell script to give it some sane defaults which works fairly well.

Yes, that's what I did for about ten years before I wrote ripgrep. :-)

> Do either of these tools have a "grep" mode, so that I can alias grep to it and get similar, but improved, behavior?

Another commenter already mentioned `rg -uuu`, and that's pretty much the right answer. In a large number of cases, if you `alias grep=rg`, then most things will continue to work as you expect. If you really do not want any kind of smart filtering, then yes, you'll want `alias grep="rg -uuu"`. But otherwise, things like `command ... | rg pattern` and `rg pattern file ...` will continue to work just like grep.

ripgrep also tries to use the same names for flags as grep, wherever possible. So you'll find a lot of overlap there too.

Please note that ripgrep is not and is not intended to be a drop-in replacement. But I didn't go out of my way to be incompatible with grep either. So a lot of it should be quite familiar!

Yeah and its also 10 times larger

https://github.com/BurntSushi/ripgrep/issues/1481

the previous issue on the topic has some pointers

Which is absolutely a good trade-off for a modern desktop system.

If you go on a system with grep aliased and run your single grep command, are you going to be able to tell which binary was larger?

If you look at the size of two disk images for different distros, and one has grep while the other has ripgrep, are you going to be able to tell which is which?

10x larger than grep is noise in 2020. Making that tradeoff in exchange for significantly improved performance seems reasonable.

Not to mention a large part of that difference is likely due to grep relying on libc, while ripgrep has to bundle its own runtime. And that's not even bringing up memory safety.

I'll take ripgrep at 10x the size of grep any day.

Yes. If that disk space is more important to you than correctness, performance and better Unicode support, then yeah, you should definitely keep using the silver searcher.

I gave more details in a response on that issue: https://github.com/BurntSushi/ripgrep/issues/1481#issuecomme...

Wouldn't you have the same problem with fd if you invoke it on unquoted globs? The problem is the semantics of the shell, not find.

Yes, but fd has implicit pattern matching without requiring the * character:

    Features:
        Convenient syntax: fd PATTERN instead of find -iname '*PATTERN*'

Seems a misfeature. Much of the time, I use find to look for files with a specific extension, which makes the trailing asterisk harmful.

    fd -e py

Also neatly bypasses the glob issue.

As another user mentioned, many POSIX and/or GNU utilities havent aged well

What do you mean by that? Users have gotten more stupid and uneducated over time? I've seen "when in doubt, escape it" in several books about UNIX and shell, and escapes and wildcard expansion are one of the most beginner lessons in a lot of other materials. However, the Internet has let everyone have a voice, for better or worse, and as a result anyone who barely knows something can now write a misguided "tutorial" about it.

In other words: don't blame the tools, nor advocate writing dumbed-down replacements which lack the composability and generality the originals had; blame the proliferation of barely-correct educational material that has spread cancerously over the Internet.

I have written hundreds of shell scripts, several which are hundreds of lines long. When I say they havent aged well, Im speaking from experience. When I use something like:

    find -name *.jpg

I know what it actually does. But what it actually doesnt isnt what it should do. The current behavior returns matched files... unless no files match. Then it returns the literal string "*.jpg". That behavior is usually going to be desired by no one. Same with:

    grep $SOMETHING

it going to fail if the variable has any spaces. Again something most people arent going to want. These both can be worked around with Shell options and "IFS" respectively, but people shouldnt have to do that. The default behavior should be sane, but in many cases the POSIX standard isnt that. For goodness sake, the "venerable" find tool doesnt even have a "maxdepth" option:

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/f...

You completely ignored my point, which is that none of this behaviour should be surprising or unexpected to anyone who has taken the time to think about how it all works together, and that this flexibility is the inherent way in which the UNIX command line is so powerful.

it going to fail if the variable has any spaces

Quoting and escaping. I mentioned that already, it's a basic part of the understanding and is not surprising at all.

Again something most people arent going to want.

You're assuming a lot. The fact that the variable substitution could expand to multiple arguments (because it's really just simple textual substitution) is in fact very powerful and useful (you can pass multiple arguments in one variable, for example), and you can always quote if you really want it to expand to one argument, the same way you would quote strings containing spaces to make them one argument. This is very consistent; observe that if $SOMETHING contains "foo bar" (without the quotes), then

    grep $SOMETHING

is exactly the same as

    grep foo bar

but what you seem to be suggesting is that it becone

    grep "foo bar"

...what? Where did those quotes come from? More importantly, how do you propose to remove them? My point is, the current behaviour is simple and consistent as well as powerful and flexible, and I suspect people are just going "it doesn't work the way I thought it would" and downvoting my comment when they really haven't thought about the perfectly logical explanation for why things are the way they are. The original developers of the shell language were most certainly not stupid.

The current behavior is not simple and consistent. If it were, then *.jpg would “expand” to the empty list of arguments, rather than remain unchanged, in the degenerate case of no match.

So, do you prefer people have this misconception instead? https://unix.stackexchange.com/q/544189/348263

Also, for the case of passing multiple arguments to a command, just use an array:

    SOMETHING=(foo bar)
    
    grep "${SOMETHING[@]}"

It will handle arguments with spaces correctly, unlike using a string variable and splitting it on spaces. There is no reason to ever use space splitting, unless you're maintaining some legacy interface where you can only pass a string instead of an array.

I almost never want the split-on-ifs-on-expansion behavior. That's one of the reasons I use zsh. When I want I can either use an array or $=SOMETHING.

The default split behavior basically forces you to always quote expansions which is tiresome. It also is confusing because almost no other programming language works like that.

And when you assigned SOMETHING you did wrote SOMETHING="foo bar" (with quotes) so it sorta makes sense visually also.

Our tools should be dead simple whenever possible rather than "if you make mistakes then git gud".

I don't understand escaping properly.

Why, on some machines (all varying distros of Ubuntu) does

    find -name *.xml

return nothing and I have to use

    find -name '*.xml'

whilst on other machines the first command returns results? The shell is always bash, it's a stock install - so why the variation?

I get that it should always be quoted, but not why it sometimes works when unquoted and other times doesn't.

Perhaps `shopt -s nullglob` has been set?

“If set, Bash allows filename patterns which match no files to expand to a null string, rather than themselves.” [1]

[1] https://www.gnu.org/software/bash/manual/html_node/The-Shopt...

> so why the variation?

Difference between dash and bash?

Shell options? `man shopt` and grep for glob

That's not going to directly solve this problem since the globs are expanded before the invoked process is even called with its arguments.

> Smart case: the search is case-insensitive by default. It switches to case-sensitive if the pattern contains an uppercase character*.

This is what I want as a default for pretty much any text search.

Emacs have had this since forever. It’s pretty neat.

Vim also provides it if you use:

  :set smartcase

Imho also anything that outputs or uses multiples of 512 bytes (blocks) by default

Sure, disks may have sectors and there may be some use cases, but we use bytes, kilobytes, kibibytes, etc now for data sizes.

In this case the shell hasn’t aged well. Besides the globing feature, the bash language is arcane and few can actually reliably write correct bash despite it being ubiquitous. Personally I would like to see a completely reimagined take on the shell.

Have you looked at powershell, scsh or fish?

shellcheck helps a ton, though.

I would argue that this case shows that the shell(s) have a problem with a feature being enabled by default. In hindsight it might have been smarter to put globbing expressions in special quotes than the other way round.

For me, Silver Searcher often fails to find matches in files that 'grep' will, easily. I've not been able to determine why this is, but it happens often enough that I consider SS to be highly unreliable, even if its fast.

So, it seems that there are still some issues to be sorted here.

This claim needs adequate proof, please provide an example. Otherwise the HN reader has to assume you made a user error.

Maybe it has something to do with my not understanding its case sensitivity rules .. which I've learned about in this thread as being different.

As soon as it happens again I'll bring it here as an example, but like I said I stopped using silversearcher because of it getting in my way like that.

Reply on news.ycombinator.com