Hacker News Reader

Comment by gargh | Hacker News Reader

Doesn't the io module (http://docs.python.org/2/library/io.html) do this without resorting to non-typical python code?

Is it non-idiomatic to do "for [line] in [file object]"? I use "for line in open('file')" all the time, for similar reasons as presented in the article. "for line in sys.stdin" is basically the same pattern, just with a different file object.

Edit: The idiom's mentioned in the Python docs on IO [1] as "memory efficient, fast, and leads to simple code"

[1] http://docs.python.org/2/tutorial/inputoutput.html#methods-o...

Nitpick: it's better to do "with open('file') as file: for line in file: ..." instead, but otherwise yes, iterating over file objects is great.

Another option is using mmap[1], particularly when the file is already in memory or you need more random access to it. It worked well when I was trying to parse some lines from the end of an open log file.

[1] http://docs.python.org/2/library/mmap.html

Is there a difference between the two? I assumed that without the "with", the file would still be closed once the loop was exited (and thus the reference to the file is dropped), but I'm open to the possibility that I'm mistaken.

Without the context manager (with statement), the underlying file is closed when the file object is garbage-collected.

In CPython, since reference counting is used for GC, this occurs when the loop exits. However, other implementations (e.g. PyPy) may use different schemes that do not guarantee collection as soon as objects go out of scope. As an extreme, a valid and occasionally useful GC strategy is to never collect anything at all [0]!

Hence, if you want to portably ensure the file is closed, you should either use the context manager or call close() explicitly.

[0] http://blogs.msdn.com/b/oldnewthing/archive/2010/08/09/10047...

Relying on GC to close the file isn't safe. See http://stackoverflow.com/questions/1834556/does-a-file-objec...

Reply on news.ycombinator.com