Hacker News Reader

Comment by galadran | Hacker News Reader

> Nowadays, whenever D is publicly evaluated, the younger languages Go and Rust are often brought up as alternatives. Here, we need not go into an in-depth comparison of these languages because both Rust and Go lack one feature that we rely on heavily: nested functions with access to variables in their enclosing scope.

Maybe I've misunderstood, but aren't Rust's closures [1] exactly what the author is describing? Specifically, you can capture variables from the environment into functions.

[1] https://doc.rust-lang.org/book/first-edition/closures.html

View on HN

Closures are, but functions defined using "fn" syntax cannot capture from the environment. This is a useful feature, and I can see why people would want it, but it was considered ultimately not consistent with Rust's aims to be excellent at the low level. See [1] for more discussion of the tradeoffs and why Rust chose not to include the feature.

[1] https://users.rust-lang.org/t/inner-functions-not-closed-ove...

I believe the story here is the same in C++. Functions are just a pointer to some compiled code. Closures are an object that represents all the captured variables, in addition to the code. That object has a size, and a destructor, and the variables it's holding might have pointer lifetimes that you need to worry about. In Rust, quite a lot of type system machinery gets invoked when you create a closure, to make sure it doesn't outlive any of the references it's closing over.

Yes, it's similar to C++. A Rust `fn()` type is similar to a C++ `(*foo)()` type and at the low level is just a pointer to code. A Rust `Fn` trait object (of which there are several variants) is roughly similar to a C++ std::function and is "fat" in that it can contain data as well as code. Most higher level languages don't make the distinction, buying conceptual simplicity at the cost of potential optimizations when a simple code pointer will do.

D nested functions don't 'capture' variables from their enclosing scope. They can access them just like other code in the enclosing function.

This is achieved by adding a hidden parameter that is a pointer to the enclosing function's stack frame. This forms a threaded list so variables in nested enclosing functions can be accessed by walking that list.

This is the same way Pascal does it.

It also means that taking the address of a nested function produces a pair - a pointer to the function, and this hidden parameter to the enclosing stack frame.

It's the same way that MetaWare implemented nested functions in its C/C++ compiler in the 1980s and 1990s, too. On top of that, MetaWare built iterators and an iterator-driven for mechanism. The yield() in an iterator was actually a nested function callback from the iterator to the body of the invoking for statement, which was turned into a nested function under the covers.

* http://jdebp.info./FGA/metaware-iterator-driven-for.html

In theory, the extra frame pointer mechanisms of the ENTER and LEAVE instructions in the Intel ISA were there for supporting this kind of nested function. MetaWare High C/C++ did not use them, preferring to use a variant calling convention where (if I recall correctly) ECX contained the extra hidden parameter.

The ENTER and LEAVE instructions only supported dynamic linking (pointer to caller's stack frame) not static linking (pointer to the enclosing function's stack frame).

That is not in fact true.

ENTER copies M words from the caller's stack frame. The pointer passed by MetaWare High C/C++ in the register could equally well have been placed at the relevant position in the caller's stack frame, just as other arguments are, and an ENTER N,1 used as part of the perilogue.

The only consideration here is speed of the ENTER instruction versus having the hidden pointer in a register, not that this sort of nested function implementation is somehow impossible with ENTER. It's just another case of ENTER is slow and other instructions do the same thing better.

* http://jdebp.info./FGA/function-perilogues.html#Standardx86

(Part of that consideration is making non-nested functions usable via full function pointers.)

You're right.

That's usually what "capture" means when talking about closures.

At least in Rust, capture granularity is per-variable, so I think closures don't contain a stack frame pointer? Though, I'm not sure why this would matter to a user evaluating Rust's closures vs. D's nested fns?

It’s a pointer to the environment, and a pointer to the function. So yes, not a stack frame pointer.

The environment is a struct containing what is captured; so for example, it would contain an &T if your closure captures a T by reference. You don’t need to walk a set of stack pointers to find it, just follow the reference directly there.

That's distinctly different from D's, which has a pointer to the enclosing stack frame.

If it is the enclosing function, it's just a single pointer to it. The "walking" comes from if you're accessing the enclosing function's enclosing function's frame.

Makes perfect sense, thanks. I’m now wondering about the trade offs of these two approaches...

Go also certainly supports this. Here's a playground example: https://play.golang.org/p/_BLguocGBRn

Does Go take a copy or a reference?

The following example will provide your answer:

https://play.golang.org/p/NeGuDahW2yP

The captured context persists between calls to the function, as the sibling example demonstrates, or this one:

https://play.golang.org/p/drfx_3g84aE

The interaction between borrow-checking and closures restricts what else you can do while you have a closure in scope that closes over some variables, so (like many rust features) they aren't a drop-in replacement for their counterpart in other langs.

The only difference I see is if you need recursive local functions, which is harder (but not impossible) to do with anonymous closures.

Recursive nested functions in D just work - the hidden parameter to the enclosing frame forms what's called a "static link". A "dynamic link" would be the usual pointer to the previous stack frame (not the enclosing stack frame).

Maybe "access to variables" means write access. I know that in Java variables referenced by a lambda need to be final.

Here's a tweak to the Go example above that demonstrates writing: https://play.golang.org/p/H9vCt93Qz2B

Here's a Rust version that does the same (without heap allocation!): https://play.rust-lang.org/?gist=bb5406062970b62088d58fb8a9f...

In Rust's case, this becomes far less practical as soon as the closure's return type does not implement the Copy trait.

Here's an example that does something similar with Vec's: https://play.rust-lang.org/?gist=7a4ed9098a53831195c5189ba9c...

I think the real nastiness happens if you want to have more than one closure over the same variables, when those variables aren't Copy. Because then when you try to `move` them in, the compiler yells at you. At that point you have to resort to putting them in an `Arc<Mutex>` or something like that. But that's par for the course with Rust: the compiler is preventing you from creating multiple mutable references to the same object.

Write access works. https://play.rust-lang.org/?gist=573c941b91657bb0dbac80c33d0...

Once "non-lexical lifetimes" lands, the inner scope won't be necessary. That may have tripped the OP up.

Reply on news.ycombinator.com