Hacker News Reader

Comment by hardware2win | Hacker News Reader

How did they even end up with such a crazy behaviour?

Technically there is no crazy behaviour, it’s a natural consequence of scoping loop variables outside the loop, which historically was common.

It became an issue as lambdas and other lambda-type constructs (which implicitly keep a reference on the loop variable) became more common, and a bunch of languages got caught in it. Later languages switched to the “inner scoping” mechanism to avoid it.

Go did not follow the switch because Go.

This is not really what's happening. The variable is scoped to the loop. The issue is that even if you take the address of the variable, each iteration of the loop will have the same variable with the same address.

    nums := []int{1, 2, 3}
    for _, num := range nums {
        fmt.Printf("%p\n", &num)
    }

This will print the same address three times. If you add "num := num" as the first line in the loop, it will print three different addresses. The proposal is to make this the default behaviour.

> This is not really what's happening.

Yes it is.

> The variable is scoped to the loop.

Maybe “loop body” would make the comment clearer?

> The issue is that even if you take the address of the variable, each iteration of the loop will have the same variable with the same address.

Hence the variable not being scoped to the loop (body), one variable is shared between all iterations of the loop.

> Maybe “loop body” would make the comment clearer?

I think "loop iteration" might.

Arguably not, because an 'iteration' is a unit of execution, not a lexical unit. If the variables were really scoped to loop iterations, that would be a form of dynamic scope, which would have a different semantics. So for example, say the loop calls a function foo. This function executes inside every iteration of the loop, but within foo, one cannot access the loop iteration variable (as it is in a different lexical context).

My point is precisely that lexical terms (like loop body) are insufficient to explain the Go behaviour. The loop variable is clearly lexically scoped to the loop body. "Scoping loop variables outside the loop" is commonly understood as the Python gotcha:

    for x in range(3):
        foo(x)
    print(x)  # prints 2

or the pre-C99 style:

    int i;
    for (i = 0; i < 3; ++i)
        foo(i)
    printf("%d", i);  // prints 3

This is not the case in Go. I don't think talking about variable scopes accurately describes the issue (because there's nothing special about loop scopes here: the same "escape" can happen from any scope), and changing "loop" to "loop body" doesn't improve this. The term "loop iteration" at least identifies the dependency between different iterations of the loop as the issue.

I don't understand what the thought experiment about non-lexical scopes has to do with this.

It's purely a question of scopes, as indicated by the equivalent code in the article:

    for _, elem := range elems {
      elem := elem
      ... &elem ...
    }

Nothing beyond regular lexical scoping and Go's ordinary assignment semantics are necessary to see how this works. The second 'elem' has a narrower scope than the first (it is limited to the loop body). Abusing Go syntax, you can think of the current semantics as follows:

    {
      var elem Elem
      for _, elem = range elems {
        ... &elem ...
      }
    }

Here 'elem' scopes outside the loop body, and so is reassigned on every iteration of the loop (and &elem evaluates to the same address on every iteration).

>the thought experiment about non-lexical scopes

It's not just a thought experiment. There are languages with dynamically scoped variables (e.g. global vars in Common Lisp).

I understood the JS version of this problem, but to see it happen with an address-of operator is just weird. In C if I took the address of a function scoped / temporary variable and kept it around it would be very bad.

I guess taking the address "promoted" the shared variable to enable it to survive past the function?

Yes, if you take a reference and it “escapes” the variable gets boxed (promoted to a pointer on the heap).

Said differently: Go puts variables on the stack only when the compiler can prove no references escape the stack frame. Stack is purely an optimization.

Except

1. Java is also supposed to do that, and clearly way less successful

2. Escape log is part of the standard baseline tools (it’s just a flag), so it’s not considered anything hidden or arcane

3. Go tends to log escapes, meaning it’s baseline is to stack allocate

So it’s a lot more than, say, common subexpression optimisation.

The subtle difference between the two ways of communicating the design: There are cases where the reference does not escape, but the compiler doesn't know how to prove that. So saying "if it escapes it's boxed" is subtly wrong.

The default is heap, and only when the compiler can be sure it's safe, things go on the stack.

It’s just what happens if you interpret the for loop as modifying a single variable binding on every loop rather than rebinding on every loop. That’s how loops have normally been conceived in imperative languages, but it doesn’t make much of a difference until you add closures (which can capture the binding) or garbage collection (allowing you to capture the address of the ‘local’ variable). The same issue existed with ‘var’ for loops in JS and bit me several times: https://stackoverflow.com/questions/750486/javascript-closur...

If you hide pointers and destructively update variables, this happens easily. I'd say the Go problem is a bit worse because there are also exposed pointers via that '&' operator.

But the problem should have been known when the language was designed. In CommonLisp, i.e., quite a mighty but old language, it is the same: the capture is on a loop variable that is destructively updated:

    (setf refs (loop for i in '(1 2 3) collect #'(lambda () i)))
    -> (#<function) #<function> #<function>)

    (mapcar #'(lambda (f) (funcall f)) refs)
    -> (3 3 3)

The same done with a fresh function parameter works. And since this is the usual style in Lisp, I suppose the problem will not be that obvious like in Go:

    (setf refs (mapcar #'(lambda (i) #'(lambda () i)) '(1 2 3)))
    -> (#<function> #<function> #<function>)

    (mapcar #'(lambda (f) (funcall f)) refs)
    -> (1 2 3)

If only this was a go specific problem:

    >>> def foo():
    ...  return [lambda: x for x in range(10)]
    ...
    >>> [x() for x in foo()]
    [9, 9, 9, 9, 9, 9, 9, 9, 9, 9]

I remember learning about this gotcha around the time that Go was coming out. I was very disappointed that they were sticking to this behaviour around the same time that JavaScript was fixing it. As a new language they get less sympathy from me.

It does feel like Go is relearning lessons of other languages quite often. It's good that this and generics could be fixed later on, but it's unfortunate that they couldn't fix the null pointers (and going further always defaulting to some default value) and error handling.

> It does feel like Go is relearning lessons of other languages quite often.

A common sentiment when Go was unveiled was that its designers had ignored the 35-odd years of language research and experience since C.

Python actually does this right for sequence comprehensions:

   >>> [f() for f in (lambda: x for x in range(10))]
   [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

List comprehensions have been around for longer, and so behave the way they do for backwards compatibility reasons.

Several languages have or have had this behaviour. C# made a similar change to its 'foreach' loops a long time ago (but not 'for' loops for some reason).

I suppose it's the natural way to do it when implementing a language and not thinking about it too much. It makes a for-loop equivalent to a simple while loop with the loop vvariable initialized outside of the loop.

In c# you mean closure capture?

Almost certainly, that is the vehicle for the issue in most languages.

To be fair it is no more difficult to implement it with the variable initialized inside of it. In fact, it's easier to.

The C# change is described in detail in the linked GitHub issue - one of C# devs left a comment explaining it.

TL;DR is that both "for" and "foreach" scoping fixes would be breaking changes, but "foreach" was easier to justify because it was already a C#-specific construct syntactically, unlike "for" which uses the same exact syntax as C, Java etc, and they were very sensitive to backwards compatibility at the time (esp. since the tooling didn't have the ability to target various language versions within the same project easily). At the same time, "foreach" represented the vast majority of breakage when they looked at existing code, perhaps because the scoping in classic "for" is more obvious due to the fact that variable mutation is explicit there.

I think it's quite normal. Because the common three part for semantic is just a suger of

    initVariable()
    while checkVariable():
       doSomethong…
       updateVariable()

So the variable is shared by all iterations at first place. And that isn't a issue until we have lambda or something similar that can capture reference of loop variable.

Later languages found it is problematic when use with reference capturing features and changed to something else (one variable per iteration)

It is a combination of scoping, which other people already wrote about, and also because the language support mutable data by default. Had the language been immutable/persistent by default, this particular semantic problem cannot occur in the first place.

Complexity increases rapidly when you combine constructs in a programming language. You get some feature interactions which are hard to get right, and also to predict.

Reply on news.ycombinator.com