Comment by dbbo | Hacker News Reader

Comment by dbbo | Hacker News Reader

Parent Full thread

To quote "The Awful Truth about sed" section of the Grymoire, "It is not your fault you don't understand sed."

I do, however, understand Perl, so that's what I use. E.g.: `echo 'fubar' | perl -lpe 's/fu/foo/'`

It might be a few milliseconds slower-- for example, commenting every line of my 352-line .zshrc (via s/^/#/) takes 0.007s total with Perl (so does `s/^/#/ unless /^\s*#/`) and 0.005s with sed. Commenting out every line of /usr/share/dict/american-english (98569 lines) takes 1.124s with Perl and 0.813s with sed.

Since I already know how to do more complicated things with Perl (like conditionals, named backreferences, etc.) it doesn't seem worth it to take the time to learn how to use sed effectively. I can wait the extra second since I'm not on any kind of deadline or under any efficiency constraints.

I am not trying to say that Perl is better than sed or any other text processing tool. I also don't mean to imply that speed is sed's only advantage-- it's just one example. I think that for someone who already knows some Perl, learning another similar tool doesn't make sense. I'm sure there are exceptions. This is only my humble, personal opinion.

For people who do need/want to learn sed, the article did a pretty good job of showing you how to get a lot done without a whole lot of reading.

On the speed question, I actually find Perl considerably faster than sed in a lot of use-cases, if there's enough processing to dominate the slightly higher startup costs of Perl.

For example, at one point I had reason to take a gigantic single-line textfile, and break it into lines based on a specific 3-letter pattern that didn't occur anywhere else:

   s/ABC/A\nC/g

In whatever sed comes with Debian, this took about 10 minutes, CPU-bound, for a 2-gigabyte file. With Perl: 1.5 minutes, IO-bound. Not too sure why. Maybe sed runs everything through the regex engine, while Perl special-cases constant strings? Perhaps Perl has better buffer management for processing gigabytes of text? I haven't done any real testing.

The copyright section of sed(1) suggests that the Debian package sed (I'm running wheezy) is Gnu sed 4.2.1[1]. Plan9 sed is also available in the 9base package. I'm running perl 5.12.4 (also from Debian testing). I don't see any significant difference in speed with vanilla 5.14.1 and the commands I mentioned earlier.

[1] http://www.gnu.org/software/sed/

Turn on locale UTF-8 for GNU software before benchmarking it to compare with Plan9 software.

GNU stuff gets a speed improvement from assuming single byte characters.

however, perl has -i, which not all seds do, so those 0.002 seconds you lost to perl will be more than gained by being able to type

      perl -pi -e 's/foo/bar/' somefile

instead of

      sed 's/foo/bar/' somefile > somefile.tmp && mv somefile.tmp somefile

Rather than -i alone, I always use -i.bak (or whatever you prefer instead of '.bak'). That way, you get (1) in-place editing and (2) a backup. The files you work on are backed up to "file.bak", so that if you make some simple regex mistake with huge negative consequences, you have a ready fix. (And if you do things perfectly, it's easy to run "rm *bak" after you confirm that.)

The utility 'sponge' is useful for doing this in the general case.

Part of the 'moreutils' package, for the curious. The description for sponge on the project homepage reads "soak up standard input and write to a file."

Sed has -i, too! Well, at least GNU sed.

yes, that's why i said "not all seds".

missed it. sorry!

because GNU is Not Unix

I know how to do exactly one thing in sed, and that's sed 's/blah/blahprime/' somefile.

If sed has capabilities other than that, I don't particularly care... but that's one thing that I need to do frequently which is more painful in awk or python.

I would guess that that's all a significant number of unix-like OS users do with sed, and in those circumstances, it does make more sense to use sed than a more extensive language. For simple substitutions like that, I'd just use an alias: alias ped='perl -lpe' to save the extra keystrokes.

Apparently the Debian kernel team agrees. I just noticed this while building a kernel package:

test -n "$k" || perl -pli~ -e 's/\$\{shlibs:Depends\}\,?//g' debian/control

Reply on news.ycombinator.com