Python for Perl programmers | Hacker News Reader

Python for Perl programmers

everythingsysadmin.com

40 pointsjoslin0137 comments

  However, most Python programmers tend to just read the
  entire file into one huge string and process it that way.
  I feel funny doing that. Having used machines with very
  limited amounts of RAM, I tend to try to keep my file 
  processing to a single line at a time. However, that 
  method is going the way of the dodo.

  contents = file('filename.txt').read()
  all_input = sys.stdin.read()

This is not the correct way to do things. The first example given is better.

  for line in file('filename.txt'):
    print line

Generators are a beautiful thing [1][2]

[1] http://stackoverflow.com/questions/519633/lazy-method-for-re...

[2] http://www.dabeaz.com/generators/

No doubt. It must be noted, though, that for a very-well-thought-of, modern language: python has a staggeringly large number of "incorrect" ways to do things.

Python has a design ethic that most other languages don't have. There are very few, if any, good synonyms for 'Pythonic.'

Though the results may have changed since, I once had to parse a very, very large file (about 5 GB) line-by-line. I tested a number of different methods, and ultimately discovered that reading the entire file using .readlines() was faster by a significant margin. I vaguely recall it was more than four times faster, actually - and that was a really big win when you're talking about a data set that large.

Granted, our machine had a perhaps-unusual design in that it had multiple RAID 6 data banks, a very powerful dedicated IO controller, and about 128 GB of RAM. So your mileage may vary.

Unfortunately I don't have access to that machine any more so I can't perform the same benchmark, but I have a nagging feeling that if I ran the tests again with Python 3.2 I'd get different results.

That's most likely because of the overhead of all those syscalls for each line.

A nicer way to do such things is to open the file with mmap, which does one system call and then just bumps a pointer. mmaps expose both a file and a list-like API.

Python people tend to always compile their regular expressions; I guess they aren't used to writing throw-away scripts like in Perl

This is so that the RE isn't compiled each time it's used. These days the RE module maintains an internal cache for compiled RE's, but that wasn't always the case.

The 1.5.2 docs for compile() say "using compile() is more efficient when the expression will be used several times in a single program" while the 2.7.2 docs add the note that "the compiled versions of the most recent patterns passed to re.match(), re.search() or re.compile() are cached, so programs that use only a few regular expressions at a time needn’t worry about compiling regular expressions."

Edit: apparently the cache was present in 1.5.2; it just wasn't documented. In 1.5.2 the cache size was 20. In 2.0 it was increased to 100.

Still: explicit is better than implicit.

As well, I find assigning the RE to a well-named variable tends to make for more readable code. RE patterns can be horribly ugly and I like to get their definitions out of the way of the logic using them.

> RE patterns can be horribly ugly

It worth to be noted that the verbose flag (re.X or re.VERBOSE in Python) may help to get more readable/friendly regexes (by allowing whitespace and comments).

I'm desperately fighting the urge to snark: but another very effective treatment for dealing with unreadable regex syntax is to learn regex syntax. In my experience probably 90% of people who complain about regular expressions simply don't know them very well.

It's a declarative language without symbols or recursion: that means that it's just not well-suited to being "broken down and simplified". You just have to bite the bullet and learn it. With a little practice, the basic features (e.g. beginning/end of line markers, character classes and captures, maybe non-capturing parentheses too) should be readable without trouble.

I think you missed my point.

I agree that for trivial regex it's useless. For instance (directly taken from Python doc) in this case:

  a = re.compile(r"""\d +  # the integral part
                     \.    # the decimal point
                     \d *  # some fractional digits""", re.X)
  b = re.compile(r"\d+\.\d*")

that's useless. But for more complex problems it may be useful (e.g. http://www.doughellmann.com/PyMOTW/re/ I know email matching regexes are not really complex or even a good example of use of regexes but that's the first example I found).

Another point is that even if I can read/write (that's the case) regexes other people may have to deal with my code and it's a well known fact that many people doesn't understand/like regex, so splitting them in small "chunks" may help them.

To be fair: my argument wasn't against the use of the /x suffix to a regex. I'm sure that there are circumstances in the wild (though quite honestly I can't recall seeing any in production code, and I've written and read a truckload of regexes over the years) where it's used productively to document a really hairy expression.

I'm just saying that 90% of the time when users complain about not being able to read a regex, the solution should be "hit the books" not "rewrite the expression".

The point being, though, that perl regexes are compiled at parse time, with no extra syntax hanging around.

And no, I simply have to disagree that a separate object instantiation is "more readable" than the straightforward syntax describing the object itself. That this argument should appear in a discussion of a dynamically bound scripting language shows as well as anything why the python community is eating itself. A LISP or Ruby nut would never have made that claim.

The point to having code there is to read it. Hiding it doesn't make it more readable, it just fools you into thinking you understand it better than you do. You probably call your regexes stuff like "split_at_word_boundaries", right? And how many bugs have you had to deal with because the whitespace conventions aren't clear from your source?

Can someone please make a Perl for Python programmers?

It actually took me 30 minutes today to figure out how to do:

text=open('file.txt').read()

without creating 3-5 lines of code which seems excessive for a simple operation.

There are apparently 20 ways to do it in Perl and they need their own CPAN library for it :-( [1]

[1] http://search.cpan.org/~drolsky/File-Slurp-9999.13/extras/sl...

http://onyxneon.com/books/modern_perl/modern_perl_a4.pdf

Perl isn't batteries included, but its packages are usually actually tested on multiple versions. This allows for the community to move on its own. There is no dictator, by design. Try #perl for help as well, or the package channel if there is one.

http://onyxneon.com/books/modern_perl/ has PDFs in other page sizes and other electronic forms too.

Yea,

my $file_handle = open(file.txt)

my $text;

while (read $file_handle, $buff, 1024) { $text = $buff }

read returns the number of characters read (0 when it's finished which will end the loop). I don't know of any generic "read" function, so it's up to you to define how many characters at a time and give the $buffer scalar to read into. it's just a different way of doing things

norkakn is a better resource than this comment

I ended up going with this one for better or worse. I'm afraid I still don't quite understand it:

my $text = do { local( @ARGV, $/ ) = $file ; <> } ;

The local $/ is the trick : It reset the line-end character to 'undef', which makes the <> line-slurp the whole file at once.

If you don't understand it, why use it? Is it that important for you to do this in one line of (relatively obscured) code, than in 2-3 lines of readable code?

For the record, the do() block defines @ARGV (which normally holds the command-line arguments) to contain the value of the $file (presumably a file name), and sets $/ (which is the input record separator) to an undefined value. In Perl, <> is synonymous to <STDIN> which makes it read from stdin. Since @ARGV now contains a file name, Perl will open this file for reading. Since $/ is undefined, Perl will enter in "slurp" mode, and will gobble up the whole file in one shot. Since this is the last statement in the do() block, it will be returned as its value, and stored in $text.

Lots of Perl magic happening here. I advise you against using such shortcuts.

my $txt = do { local $/ = <$file>; };

my $txt = do { local $/; <$file>; };

are as I know the preferred ways to do it. As a previous comment explained, $/ is the record separator and when you localize it it's value is set to undef. Which means we read the whole filehandle (the second part). As a do block evaluates to the value of the last expression, the contents of the file is the result of the whole expression.

If you're serious about Perl I recommend Modern Perl which is a 250 page really good introduction for the language, intended for programmers.

Agreed, the trick with a local ARGV and the null filehandle is too clever. Don't copy code you don't understand! I think opening the file is much clearer.

    open( my $fh, '<', $file ) or die "cannot open $file\n";
    my $text = do { local( $/ ) ; <$fh> } ;

  http://perldoc.perl.org/perlopentut.html
  http://perldoc.perl.org/functions/do.html
  http://perldoc.perl.org/functions/local.html
  http://perldoc.perl.org/perlop.html#I%2fO-Operators

CPAN's ecosystem of packages are not really like Python's Libraries. There are packages for very small jobs such as doing IO or processing command line options (getopts) as well as packages that serve as whole ORMs or web frameworks, so it's considered easy to import File::Slurp even if you've never used it before, whereas it seems like the idea of using a library on the scale of Scipy for something like that would be silly.

My understanding is that Python usually has one established way of doing something, and what you find when you search "Python foo", what you find is usually what you want. Perl has more of a market based approach. I don't think "There Is More Than One Way To Do It" is intended to encourage the reinvention of wheels, that isn't really the Perl way. There often is an established way to do it in Perl and that way is File::Slurp or Template::Toolkit or Moose or DBI::DBD. The packages are great, they have maintainers so they can be updated, and as long as you know to use CPAN, they are easy to find.

On the other hand, that means the difference between a beginner who has problem X and says "Oh I'll look on CPAN for the source of the most popular package for X and see the examples the author put in" and a beginner who says "I don't know how to do that in Perl, where's a tutorial online?" is huge.

A couple of weeks ago there was a big to do in the Perl community that there needed to be more well-written up-to-date results for "how-to" searches. I think the reason that the community got into that situation is that for people who know to use CPAN, the quality curation is taken care of. It is always clear what the top 1 or top 3 packages are for a given use and the place to start is looking at the examples in that package's documentation.

TL;DR : For Perl, CPAN's packages serve as 'reuseable code' as well as ORMs, and web frameworks etc, whereas it seems to me that Python will port most pieces of reusable code into the main language itself.

Edited: Replaced "I don't need to do that in Perl" with "I don't know how to do that in Perl"

Right. When looking at the core language, "There Is More Than One Way To Do It" becomes "More than one thing will compile to the same VM opcodes, so you can choose the one that best expresses your intent to the maintenance programmer."

When looking at libraries, "There Is More Than One Way To Do It" becomes "More than one library will solve your problem and you're welcome to select the one that makes the speed/size/complexity/feature trade-offs that best fit the project at hand."

This is, absolutely, initially more confusing than "there should be one obvious way to do it", but it has other advantages, especially in the long run.

This is actually really simple in perl. The idiomatic way to do it would be:

  open(my $fh, "<", "file.txt") or die "wft?"
  my @lines = <$fh>;

lines is now an array of lines, you can join them into a single string if you want.

Slightly more idiomatic would be with a useful die (and a semicolon terminating the first line):

  open(my $fh, "<", "file.txt") or die "Error reading file: $!";
  my @lines = <$fh>;

To slurp into a string rather than an array of lines:

  open(my $fh, "<", "file.txt") or die "Error reading file: $!";
  my $s = join('',<$fh>);

Or just:

  my $data = `cat $file`;

Some folks dislike the useful use of cat, but anyone who's ever read a shell script will recognize the idiom. And it's a trivial one-liner much simpler still than the python code .

This isn't exactly idiomatic Perl and fails completely in environments that don't have cat (Windows).

For one-time uses there's not a problem. However, if you find yourself writing what amount to shell scripts in Perl, you might be better off just writing shell scripts.

Excuse me? Who are you to determine what is and isn't "idiomatic"? And that you would try this kind of thing about perl of all languages is just shocking. What's saddest is that it's the lack of this kind of absurdist pursuit of robustness and purity is one of the things that has always best defined the perl community.

Broadly, your argument seems to be that spawning processes in the external environment[1] (which, admittedly, is inherently nonportable) is a Bad Thing in perl, and that we shouldn't do it. When I rephrase it like that, does it sound as poo-flinging crazy to you as it does to me?

[1] Let's be honest: a working "cat" is pretty much the single most portable thing you can put between the backticks. If you won't allow this, what will you permit?

First off, apologies if I was coming off as rude, I wasn't trying to be.

My point was more that slurping a file without spawning another process is already quite easy in perl. That method of slurping contradicts common patterns like "while (<$fh>)". I'm not saying it's strictly wrong or even bad, just not necassarily the best option.

This post single handedly uncovered an idea for a new niche' of premium programming books: how to program in language A for language B programmers.

Sorry can't use Python instead of Perl. Basically because Python is not Perl.

Now if you want to just change a language for the sake of changing it, its Ok. But Python and Perl have opposite philosophies. And the actual problem while changing from Python to Perl occurs there.

To me Python looked like Java with a haircut and french beard. That fact simply keeps hovering over me, even in this tutorial all those variable = something.somethingElse.someMethod() thing gives a strong Java vibe .When I used it for the first time, It looked like a language for people who were frustrated with Java but at the same couldn't learn Perl either. It looked like a language designed for such people.

Parsing a program visually hurts my eyes. Especially for large code blocks. Inability to provide easy ways of writing throwaway command line hacks is just not acceptable at all on Unix machine. That's precisely where Perl won over sysadmins during its early days.

Having to bury each statement you suspect under piles of try/catch statements doesn't feel like a scripting language. No multi line lambdas is a major turn off.

Regular expression support looks totally alien and its no way closer to Perl's way $line =~ /<match something>/ and not just that, Matching is just one part. Extraction, substitution et all all part of regular expression operations. Now regexes combined with map/grep functions make parsing a whole lot easier in Perl.

Scoping sucks big time in Python. There is nothing remotely comparable to CPAN.

I don't know which versions to Program in, 2.x or 3.x? There is nothing like Moose and other associated modern perl packages in Python.

Lastly I never needed another C based language for scripting. What I needed was a more extensible language(like Perl6).

Python doesn't serve my scripting needs on Unix. But I agree it has good web frameworks and it may be helpful there.

Here is a good "cheat sheet" for how to do X in Y (where X and Y are in PHP, Perl, Python or Ruby): http://hyperpolyglot.org/scripting

It doesn't cover idioms like Tom's post, but helps me remember the basic syntax changes when jumping between these languages.

Anyone have the opposite? I've been working in Python for a few years but now I'm taking a Perl class.

I've not used Perl in any real way for years but I am now brushing up on it again.

You can do neat one-liners in Python or Ruby but I've always liked Perl for this kind of thing.

I also have a weird penchant for using BASH scripting to do quite complicated things. Just because it can.

Not mentioned is the common idiom of assigning regexp matches.

In perl,

> my ($a, $b) = '12' =~ /(\d)(\d)/;

In python,

> (a, b) = re.match('(\d)(\d)', '12').groups()

Note the python code here will throw an AttributeError if the match fails, whereas the perl will just assign undef to the variables.

Alternatively, use named captures:

  > obj = re.match('(?P<id>\d\d)(?P<name>\w+)')  
  > id = obj.group('id')  
  > name = obj.group('name')

Nice!

Actually, perl will return an empty list, which if then evaluated in scalar context will return 0, which is boolean false. So

  if (my ($x, $y) = '12' =~ /(\d)(\d)/) {
    <use $x and $y here>
  }

is the normal idiom.

Reply on news.ycombinator.com