Hacker News Reader

Comment by mjw | Hacker News Reader

Quick question for anyone here familiar with core.async:

Would it be possible (and if so what would be the simplest way) to implement something like Python's generators and `yield` statement in Clojure using core.async? I'm thinking something like:

    (defn range [n]
      (generator
        (loop [i 0]
          (yield i)
          (when (< i n)
            (recur (inc i)))

    (let [generator (range 5)]
      (generator) ;; => 0
      (generator) ;; => 1
      ;; etc
      )

View on HN

I dunno, but I can do it in go :-P

  func generator(values ...interface{}) func() interface{} {
    c := make(chan interface{}, len(values))
    for _, v := range values {
      c <- v
    }
    return func() interface{} { return <-c }
  }

"The problem with using a goroutine as a generator is that if you abandon it, it will not get garbage collected."

It's also slow.

https://groups.google.com/forum/#!topic/golang-nuts/v6m86sTR...

My design relies on pre-loading the channel's buffer and doesn't use a goroutine.

You could presumably do this with core.async, but the more idiomatic solution to generators would be lazy sequences.

Oh for sure, more as an exercise in curiosity than anything else. (Although I am also playing with generator- and iterator-like abstractions as part of a resource-scoped foldable stream abstraction to help process big files.)

Looks like I managed to answer the 'is it possible?' part of the question anyway -- something like this:

    (defn range
      [n]
      (let [c (chan)]
        (go
         (loop [i 0]
           (>! c i)
           (if (< i n)
             (recur (inc i))
             (close! c))))
        (fn [] (<!! c))))

Yes, but I wouldn't do that. Every time you pull an item out of the channel, you actually submit a task to execute in a separate thread pool (to produce the next number), block your own thread, and wait for the task to complete on its thread and then wake your thread up. That's a lot of task-switching going on just to generate the next consecutive number :)

OTOH, you could use async's coroutine code (used to implement go blocks) to create generators, but because you have lazy sequences, that's not necessary either.

Yep I figured this would probably not be very efficient.

About lazy sequences: sometimes you want to avoid the allocation of lots of intermediate cons cells when mapping/filtering/etc. The reducers framework for example manages to avoid this sort of cost when the data structure supports something faster than first/rest recursion. I'm interested in extending reducers to work nicely over large files and other sequences which don't fit in memory, and some kind of iterable or generator-like abstraction could play a useful role in this.

(Actual coroutine-based generators might not be necessary, but would allow for a neat outward-facing API)

you could do it with channels, but also by extending the go macro itself. The mini compiler behind the go macro is designed to be extensible. In the test suite for core.async the compiler is run through a series of tests using a "runner" implementation of the go macro: https://github.com/clojure/core.async/blob/master/src/test/c...

I've had generators working several times with an approach like this, but as it doesn't really fit with the rest of the library, I removed them.

All that being said, these APIs are internal and could change at any moment.

Aha, thanks for the pointers, that helps. I suspected something simpler might be possible using these ioc macros but was frankly a bit scared off by them at first :)

It'd be nice if the coroutine/inversion-of-control stuff was given a stable public API at some point, because those macros seem very powerful and neat in themselves and might have other interesting uses aside from channels.

Reply on news.ycombinator.com