Hacker News Reader

Comment by benhoyt | Hacker News Reader

I'm an experienced Go programmer and still accidentally do this every so often. It certainly bit me and my team several times when we were newish to Go. And particularly for newbies, it's hard to debug and understand when you do run into it. All of which to say I'm really glad they're trying to fix this.

View on HN

I've been annoyed more than once by Java's “local variables referenced from a lambda expression must be final or effectively final” error. I used to suspect that this was just an excuse to simplify the javac implementation, but I'm not so sure anymore. It is interesting to see the issue from the other side.

The issue is not restricted to variables declared in loop headers, so the proposed loop change for Go might only be the start.

> I used to suspect that this was just an excuse to simplify the javac implementation, but I'm not so sure anymore.

AFAICT it doesn't simplify javac much, if at all. It still needs to synthesize closure objects with fields to store the closed-over values. It's just that those fields can be final.

I think Java did this to avoid programmer confusion. I think it was the right choice.

It simplifies the implementation immensely, because the fields in those synthetic objects can be populated with copies of the variables from the stack frame. This:

    void foo() {
        int a = 1;
        Runnable r = () -> System.out.println(a);
        r.run(); // prints 1
    }

Can get turned into something like this:

    class r_closure implements Runnable {
        final int a;

        r_closure(int a) { this.a = a; }

        @Override
        public void run() { System.out.println(a); }
    }

    void foo() {
        int a = 1;
        Runnable r = new r_closure(a);
        r.run(); // prints 1
    }

The local a is a perfectly normal local, and the field a is a perfectly normal field.

What would have to happen if the variable was mutable? For example, if you wanted to write this:

    void foo() {
        int a = 1;
        Runnable r = () -> System.out.println(a);
        a = 2;
        r.run(); // prints 2
    }

You have to transform it to something like this:

    class r_closure implements Runnable {
        int a;

        r_closure(int a) { this.a = a; }

        @Override
        public void run() { System.out.println(a); }
    }

    void foo() {
        int _a = 1;
        r_closure r = new r_closure(_a);
        r.a = 2;
        r.run();
    }

Where there is no local, and where the method looks like it's accessing a local, it's actually reaching into the closure and mutating its field!

Now think about doing this if you've captured a variable in two closures, or a variable number of closures in a list. The wheels come off this approach.

Instead, you would have to promote the shared mutable variable to its own object, like this:

    class int_box {
        int i;

        int_box(int i) { this.i = i; }
    }

    class q_closure implements Runnable {
        final int_box a;

        q_closure(int_box a) { this.a = a; }

        @Override
        public void run() { System.out.println("q = " + a.i); }
    }

    class r_closure implements Runnable {
        final int_box a;

        r_closure(int_box a) { this.a = a; }

        @Override
        public void run() { System.out.println("r = " + a.i); }
    }

    void foo() {
        int_box a = new int_box(1);
        Runnable q = new q_closure(a);
        Runnable r = new r_closure(a);
        a.i = 2;
        q.run(); // prints 2
        r.run(); // prints 2
    }

Now you've taken a simple local variable which just need to be copied, and turned into its own thing on the heap!

It gets even worse. In the general case, you need to use a separate object for each captured mutable variable for space safety reasons (avoiding unwanted object retention not present in the source code). The easiest way to achieve space safety involves a separate object for each captured variable, but of course that is unnecessarily wasteful in many cases. But in order to coalesce captured locals into fewer reference objects, you need to do some sort of lifetime analysis to find cases where locals become unused at the same time.

> The issue is not restricted to variables declared in loop headers, so the proposed loop change for Go might only be the start.

Technically it’s not but practically it’s by far the most common way for this to unexpectedly arise.

The other cases like closing over a variable and then modifying it before the closure is invoked are a lot less common to hit unexpectedly, and a lot harder to fix nicely (short of Java’s big hammer).

Yep, bitten by this one on repeat myself. definitely a WTF moment each time.

same here, happened several times when launching a goroutine inside the loop

the fix was usually adding x := x

before go func() { do something with x }

A common alternative fix is to use IIFE, since the expression must be a call.

`go` evaluates everything but the final function call in the context of the caller. So

    go func (x int) { … } (x)

Will do the same, and is easier to extract to a named function.

The latter is also easier to re-inline accidentally because

    go func (x int) { do_work(x) } (x)

seems like a very roundabout way to say

   go do_work(x)

The "x := x" solution too suffers from this problem but slightly less: both idioms look like they are no-ops (while they are actually not) but at least "x := x" is weird enough to look like it was a deliberate choice, not some vestige from refactoring.

Er… it makes no difference?

Your first version is in fact a worse way to write the second one.

Huh. Now the closure-capturing in Go makes even less to me: instead of capturing the variable's current value it captures the variable itself i.e. puts &x into the closure instead of x — and no other piece of Go does that, although I was sure "go fun(args)" passed args by-ref but apparently not.

What's even the point of capturing the variable itself? To allow for writing inline callbacks that could sneakily mutate loop-local variables?

More generally allow the closure to write to its environment. That is the normal behaviour of closures in imperative langages, Java being the major exception because it rejects closing over non-final variable (and obviously that only blocks assignment, if the object is mutable you can do what you want to it).

In c++ close by value/reference is under programmer control and there is no default

What would be the use of closing over the value rather than the variable? That would stop a lot of interesting use of closed-over variables (like persisting values from call to call).

In fairness while that would be a lot less convenient in reference-based langages in Go you could just close over a pointer to the variable, making the relationship explicit.

That’s how you’d do it using a [=] lambda in c++ or a move closure in rust.

rsc himself mentioned having been bitten by it on quite a few occasions, too.

It's even worse in python where loops do not even create a new scope and assign on each iteration. From reading this Go creates a scope for the entire loop, but assigns rather than initializes on each iteration.

Then again, Python has the same syntax for assignment and initialization.

Reply on news.ycombinator.com