Hacker News Reader

Comment by pfranz | Hacker News Reader

I think they're really good to be aware of, but think it's overreaching to advise "Use slots when defining a Python class."

I'm surprised there's no mention of exceptions. Constructing, throwing, catching, and discarding an exception can be relatively slow (especially in a tight loop). My usual advice is "exceptions should be the exceptional case."

In general, get familiar with inspection tools so your code is easy to measure so you can clean up hotspots. Trying to optimize code without inspecting it often makes code harder to read and may not address the slower-performing parts. Maybe everyone needs to spend hours trying to speed things up just to realize the slow parts were in a different part of the code and now it's harder for the next guy to reason what's going on.

View on HN

One of the best tools for profiling I've used with python is gprof2dot, which analizes cProfiles output and generates a function call-graph that gives deep insights as to what is really taking execution time.

It can help you see a lot clearer why things are slow. Though it has its nuissances, it significantly improves your understanding of where time is spent. It's the same data from cProfile, but much better organized.

That tool, combined with the line_profiler (kernprof) can work wonders.

I made a wrapper of cProfile and gprof2dot, which can be found here:

http://bprofile.readthedocs.io

so that you don't have to remember how to call gprof2dot all the time.

Nice, it looks handy!

Slots have benefits outside of the memory savings. By preventing arbitrary attribute assignment on an object, slots can provide nice guarantees about the shape of an object, making it easier to reason about.

> exception[s] can be relatively slow

That really depends on your hit rate. Say you have a generator returning objects and you don't know if they have an attribute. You can use hasattr(..) to check or just try to use the attribute and catch the AttributeError.

One-on-one the exception is slower, but there comes a point where the exception is rare enough that actively checking every single item is slower. Eg if only 1-in-1000 doesn't have the attribute, just mop up with an exception.

Out of curiosity, is there any overhead to the python exception mechanism if you don't hit an exception (ie just by wrapping things in a try block)

I come from an embedded background, where in some c++ projects, we would disable exceptions for various reasons, including the memory overhead, which is why I ask..

Exception handlers in CPython work by having an instruction store the offset to the start of the handler code which is stored on a stack attached to the frame (iirc). So the cost of a "try:" block is pretty low (for Python, anyway).

The handlers themselves are pretty basic, an "except SomeError" pretty much translates to "if isinstance(exc, SomeError):".

That's what I was trying to get at. Not to avoid exceptions, but to write your code in such a way that exceptions are rare. I usually encounter it with file i/o situations, like like you're saying, an extra file stat in a loop would be really slow and open you to race conditions (file being modified before testing an acting). If an exception was thrown 90% of the time, you want to rethink your logic because performance might be equally poor. Catching the rare exception is ideal.

As an aside (since you mentioned generators), I wish exception handling in comprehensions were easier to deal with.

Reply on news.ycombinator.com