I've mainly used Java which doesn't suffer from this particular problem. But I've often heard Java criticized on the grounds it captures values not variables, and that most other languages apparently capture variables. I remember learning Lisp at uni and being taught that it captured variables, for example.
So do other languages suffer from this problem as well? How do they deal with it?
1. the interaction between mutable locations and closures
In most languages with mutable bindings, if you close over a variable you're closing over a "cell" and will see future modifications to it. This is the case in Go, Python, Ruby, Javascript, C#, ...
A few languages avoid it e.g.
- Java, not because it "captures values" but because it only allows capturing "final" variables so you can't update the bindings (you can still mutate in place when capturing reference types)
- C++, because you have to specify the capture mode and it'll only be affected if you capture by reference (which you wouldn't do for a loop variable), so the likelihood of hitting this issue is pretty low
- Rust, because even if you capture by reference you can't modify a binding with an outstanding reference
2. the scoping of loop variables, if you're capturing the loop variable but each iteration has a different version of the loop variable then you don't really mind too much, because each closure will capture its own loop variable, this doesn't fix the above issue but it does fix the most common occurrence of it
That's what Go is doing, other languages which have done that are C# (when using a `foreach` loop), Javascript (when using `let` or `const` for the loop variable), ...
Also Ruby, kind-of: using the `each` method for iteration is very common, and since then the loop variable is a parameter to the block it's basically a per-iteration local, "for...in" still has the issue. I guess C# also has that when using the linq ForEach method but I don't know how common that is.
I think Go is doing the right thing here. I'm somewhat surprised they didn't always do this. (Dart has created a new loop variable for each iteration all the way back since before 1.0.)
In Lisp, you can easily write macros that have whatever semantics you want. You can easily write a my-dolist which is like dolist but freshly binds the item variable for every iteration, and next to it write a my-dolist2 which assigns a single variable.
In languages like Go, the providers of your language provide all the syntax from behind a wall, and make these decisions for you. Knights in shining armor duke it out on the rooftop of an ivory tower, while lower vassals blog incessantly about the important work being done and decisions being made.
By the way, MacCarthy's ancient Lisp (Lisp 1, Lisp 1.5) didn't have lexical scope, so it would not have made any difference whether each iteration stepped the variable with setq or bound it with let. There would just be the one dynamic variable in all cases, not captured by any lexical closures (those being nonexistent). You'd want to bind it at least once with let so the prior value is restored when the loop is done.
In Common Lisp, if we use a special variable as the loop variable, it likewise won't make a difference whether it is stepped or bound each time, since the binding isn't lexical.
That, combined with the fact that things are often poorly documented is what IMHO ruins the language ecosystem.
I want to like Lisp and I love its interactivity but I find the language a bit more "construction material" language than a ready-to-use immediately language. What is bad about that is exactly what you mention: everyone will come up with a set of macros that others cannot fully grasp.
If you have to reverse engineer someone's macros, you have all of the following going for you
The macros run at compile time and then go away (unless the program is based on a dynamic compiling paradigm). They have no behaviors at run-time; their generated code does. No matter how complicated the macro, you can just run it and capture the output code, and through multiple examples get an understanding of what it's doing.
If someone writes a buggered function, you may have to end up debugging it on a target system, perhaps an embedded one.
The sky is the limit there. If it's a function in a kernel, you may have to debug some race condition between it, some interrupt handler and a piece of hardware.
No such thing ever happens when debugging a macro itself.
A macro doesn't interact with complex application state, because that doesn't exist. You never have to attach some gigabyte database and get some objects into the right state before reproducing some expansion problem in a macro. Code generated by a macro could be involved in a problem like that, possibly as a root cause, but tracing through the macro itself isn't.
The people who write awful macros are usually not smart enough to do anything that is actually hard to understand. The problems you will find are lack of hygiene, multiple evaluations and such.
Lisp makes this so easy to write macros and they have been promoted to be the "my-powerful-dsl-is-the-right-way-to-solve-a-problem" that, IMHO, it works actively against team work unless it is really well-dcumented. Now, mix that with the fact that code is not usually deeply documented and you get a cultural problem into the whole ecosystem.
> A macro doesn't interact with complex application state, because that doesn't exist. You never have to attach some gigabyte database and get some objects into the right state before reproducing some expansion problem in a macro. Code generated by a macro could be involved in a problem like that, possibly as a root cause, but tracing through the macro itself isn't
Tracing the macro can be as difficult as its expansion. If it gets obtuse it will add a lot of time to your workflow. I know they are powerful but it is probably the last feature I would use in a team, with a lot of care, and for trivial things or for really justified cases and with good documentation. If your code is full of macros, forget about readability, you threw it out through the window.
> The people who write awful macros are usually not smart enough to do anything that is actually hard to understand. The problems you will find are lack of hygiene, multiple evaluations and such
Because writing macros is not that simple in the first place. I find much easier to write regular code and use the well-known macros than grasp anonymous macro code. There are things you can only do with macros though, like lazy evaluation. But macros must be managed with extra care. For example, maybe instead of a macro it is better to use a higher order function with a closure to keep the code understandable than burying things in layers of macros.
You have to get your talking points all in a row.
For example, what does the following Python code print?
funcs = list()
for i in range(10):
funcs.append(lambda: print(i))
for f in funcs:
f()Java forces you to final copy the variable.
let funcs = [];
;(() => {
let i;
for (i = 0; i < 10; i++) {
funcs.push(function() { console.log('i=',i); })
}
})()
for (let f of funcs) {
f.call(null);
}
Specifically, I think, any language with closures where the loop variable exists in a scope enclosing all loop iterations with a value that is updated each iteration rather than a fresh variable in a scope local to each loop iteration has this problem.
In Ruby this wouldn't affect idiomatic looping methods like #each, but would seem likely to affect the less-idiomatic for loop, which is built on #each but has the control variable in the surrounding scope. (Not 100% certain how it desugars though.)
I don't know how it desugars, but it is indeed affected when using `for...in`
funcs = list()
for i in range(10):
funcs.append(lambda: print(i))
del i
for f in funcs:
f()
funcs.append(lambda i=i: print(i))Maybe this example where it returns the closures from a function is convincing?
def fs():
funcs = list()
for i in range(10):
funcs.append(lambda: print(i))
return funcs
for f in fs():
f()