Line-by-line memory usage of a Python program | Hacker News Reader

Line-by-line memory usage of a Python program

fseoane.net

130 pointsvgnet10 comments

Pretty sure forking a process every line is not the most efficient way to do this :) But nether the less, very cool

There are more efficient ways, but they are platform specific. Using ps is easier, as it's standard.

Not on my machine. :-)

   C:\>ps
   'ps' is not recognized as an internal or external command,
   operable program or batch file.

The guy why suggested using the psutil is onto something -- definitely a 3rd-party dependency, but rolls all the platform specific messiness into a library. And much faster and less fragile than forking a new process.

This reads more like proof-of-concept. There are already 2 forks on github that use faster ways.

Cool. I didn't know that you could get a representation of a function's code through func_name.func_code. That's pretty slick!

I learned that from Robert Kern's http://packages.python.org/line_profiler/

You can not only get it, you can manipulate it at runtime as you wish. I may recommend checking out the python bytecode format, its pretty straightforward. See the module 'dis' for more.

Yeap. I find it great for prototyping: I code in the IPython shell, and then I dump the code of the various functions I wrote to a file, using func_code.

    def makedumper(filen):
        def dumpfunc(func):
            with open(filen, 'a') as f:
                file.write("\n\n{}".format(func.func_code))
        return dumpfunc

     #then to use it:  
     dump = makedumper("sourcefile")
     dump(func1)
     dump(func2)

Really neat solution, useful for straightforward things. However, it's quite far from telling us the full truth. Python won't release memory instantly and there will be some free memory that can be reused but won't ever be reused (int arrays?)

Another Python memory profiler: http://guppy-pe.sourceforge.net/#Heapy

Reply on news.ycombinator.com