Comment by kazinator | Hacker News Reader

Comment by kazinator | Hacker News Reader

Parent Full thread

I've written less than 1000 lines of Python in total probably, but I correctly spotted the problem.

Python has this misfeature whereby it didn't correctly crib Common Lisp's evaluation strategy for the expressions that give default values to optional function arguments.

When you have an argument like foo=obj.whatever() the obj.whatever() is evaluated (would you believe it!) at the time the definition of the function is being processed, not at the time when the function is being called and the value is needed.

Is suspect this was done on purpose, for efficiency. Python has another misfeature: it has no literal syntax for certain common objects like lists. There is [1, 2, 3], but that is a constructor and not a literal: it has to create a new list every time it is evaluated and stuff it with 1, 2, 3. (Unless a clever compiler can prove that this can be optimized away without harm.)

The designer didn't want a parameter like list=[] to have to construct a new, empty list object each time the argument is omitted. In Lisp '(1 2 3) and '() are true literals. Whenever they are referenced, they denote the same object. The programmer has a choice here: they can use (list 1 2 3) as the default value expression or '(1 2 3). The former is like [1, 2, 3]: it yields a new object each time that is mutable; the other will (almost certainly) yield the same object and cannot be reliably, portably modified.

Hey, modern popular languages have most of the features of Lisp, so you're not missing anything.

While you are correct, that is not what is happening in the blog post. Their issue was inside a class definition, not a function.

> When you have an argument like foo=obj.whatever(), the obj.whatever() is evaluated at the time the definition of the function is being processed, not at the time when the function is being called.

This can't be correct, surely? What if .whatever() relies on internal state that changes after obj is initialized (or after the function surrounding foo is declared, not sure what you're saying)?

It is correct, it's one of the most surprising things about Python and it causes a number of mistakes, even for experts.

The easiest way to see this is by running something like this and seeing what gets printed out and when:

    print("1. start")
    
    def function(arg=print("2. func definition")):
        print("4. func call")
    
    print("3. after definition")

    function()
    function()
    function()

You should see that the print statement in the default position is called once, when the function definition itself is being evaluated, not when the function gets called.

My canonical example is that you want a function to have an argument that defaults to an empty list:

    def func(arg=[]):
        arg.append(1)
        print(arg)
    func()
    func()
    func()

shoving a print into a function definition is weird and not something you'd do normally. But someone who doesn't know this footgun is going to write a function that defaults to an empty list, and then tear their hair out when things are broken.

That's true, this is the example that most people will run into in practice. But the print example is useful because it shows what's going on more explicitly: the default argument gets evaluated as part of the function's definition. I think this helps people get a better intuition for how Python's interpreter works when it comes to evaluating function definitions.

I find this fascinating because while I don't write much python, this is the behaviour I would assume to be correct based on everything else I've seen. I wouldn't expect it to be evaluated on every call. Definitely shows that your previous familiarity can trip you up (or not)

> it's one of the most surprising things about Python and it causes a number of mistakes, even for experts.

Except every python 101 text seems to go over it, and people seem to have suddenly forgotten about it

ChatGPT driven development maybe?

This has been a source of bugs since long before ChatGPT. I suspect Python tutorials mention it so often because it's such a surprising feature and causes so much confusion. I've been using Python for fifteen years, and it's the sort of thing that I'll still absent-mindedly forget if I'm not careful.

Ironically, in C++ - which does it the other way around - the tutorials mention that often because it is also such a surprising feature (i.e. people do not expect evaluation to occur every time there).

Thing is, there's no obviously correct behavior here, and there are valid arguments to be made either way. Which is why many languages dodge the bullet by only allowing for compile-time constants as defaults in that context (and if you want to evaluate something at runtime, you can always use an optional and do an explicit check inside the body, or provide an overload).

And because it is evaluated once, unlike in say TypeScript, when you modify `arg` in the first call, the results are visible in the second call.

https://stackoverflow.com/questions/1132941/the-mutable-defa... and https://www.valentinog.com/blog/tirl-python-default-argument...

basically, having a default argument value in a function definition means to evaluate it during definition time of that function, not when the function is invoked.

This is a foot gun.

if the default argument is an object it’s reused between invocations. Hence why setting default parameters to empty list / empty dict is flagged by static analysis suites

There is an obj visible at the point of the function definition. When that function definition takes place, obj.whatever() is called, and the value stashed away. Thus the call obj.whatever() has to work at that time. The stashed value is retrieved whenever the corresponding function argument is missing.

Good explanation...

Reply on news.ycombinator.com