Python's "yield" - In Scheme (Using Continuations)

to be fair, anything you can do with call/cc (or even delimited continuations, or whatever) can be done with closures, its just a matter of being willing to write all of your code in the continuation passing style :-). Please see the linked article ftp://ftp.cs.cmu.edu/user/jcr/histcont.pdf or wikipedia for further reading.

A closer equivalent in perl, and other similar languages with closures, is the manually-unrolled continuation:

    sub continuationish {
        my $args = shift;
        my $state = {args => $args, state => 'INIT'};

        return sub {
            goto $state->{state};

            INIT: {
                # initial setup...
                $state->{state} = 'NORMAL';
                return $first_output;
            }

            NORMAL: {
                # do stuff
                if (!$almost_done) {
                    return $next_thingy;
                } else {
                    $state->{state} = 'TERMINATING';
                    return $next_thingy;
                }
            }

            TERMINATING: {
                $state->{state} = 'TERMINATED';
                return $state->{last_thingy}; # set above somewhere
            }

            TERMINATED: {
                return $sentinal_value;
            }
        };
    }

Everywhere you'd use a variable, you have to use something out of $state. This works. Any function can be rewritten using this approach to become a "generator" in Perl or similar languages. (Just remember that for and while loops all ultimately compile to "goto"s, and just perform the transform yourself.) But... you'll come to appreciate continuations or Python's yield pretty quickly. I think there have been three functions in my ~10 years of perl programming where I deemed it worth jumping through these hoops. Nice tool, but if you're using it a lot something's wrong.

There exist some modules that try to do this automatically, however I have had some trouble with them, and they have the major problem that they end up being relatively opaque, often involving things like source code filters. Also, the module documentation can feed you bad ideas about this enabling multithreading or being a good idea for routine use; in fact it doesn't affect threading at all and I'd run screaming from any module written by an author confused enough to think it's related even tangentially to multithreading. Since you really shouldn't be doing this all the time, I actually recommend doing it manually in a Perl-like language, or switching to a language that has much better support for this style if at all possible. YMMV.

> it requires the list of return values to be precomputed.

I thought Scheme's semantics were allowed to be lazy to some extent, but I'm unaware of the finer points of it. Would a conforming implementation allow being lazy in this instance?

I definitely agree that this seems like a strange place to pull out continuations. I get the impression that the author started from wanting an example of continuations, rather than an implementation of yield. As other people have pointed out, continuations and closures are equivalently powerful given CPS, but continuations seem simply gratuitous for this example.

Just to make my point, here's my Common Lisp implementation:

  (defun generator (list last-val)
    (lambda ()
      (if list (pop list) last-val)))

Scheme would have to roll its own Pop macro, but I don't see how that would require a raw continuation primitive.

Isn't it the very idea of the yield "command" that you don't have to generate a list in advance but create it's values as needed? Hence, a statement like "you don't need ... but you have to ..." sounds funny to me.

actually it goes back even farther!

ftp://ftp.cs.cmu.edu/user/jcr/histcont.pdf has a history written by John Reynolds, probably the world's most senior (and baller in the sense of doing amazing work) among currently active programming language theoreticians.

Quote from pages 2-3 of the linked pdf:

Apparently, the earliest description of a use of continuations was given by Adriaan van Wijngaarden (Director of the Mathematisch Centrum in Amsterdam) in September 1964, at an IFIP Working Conference on For- mal Language Description Languages held in Baden bei Wien, Austria. A written version of this talk, along with a transcript of the discussion that followed, appears in the conference proceedings [45].

Van Wijngaarden’s goal was to formulate a preprocessor that would translate Algol 60 into a more restricted sublanguage. The final stage of the preprocessing was (what we would now call) a transformation of proper procedures into continuation-passing style (CPS) [41], with an at- tendant elimination of labels and goto statements. (An earlier stage of the preprocessing replaced function procedures by proper procedures.) As van Wijngaarden described the transformation:

Provide each procedure declaration with an extra formal param- eter — specified label — and insert at the end of its body a goto statement leading to that formal parameter. Correspond- ingly, label the statement following a procedure statement, if not labeled already, and provide that label as the correspond- ing extra actual parameter. [45]