Composable Pipelines [in Go] Improved
blog.gopheracademy.com
blog.gopheracademy.com
[0]: https://ocaml.janestreet.com/ocaml-core/111.28.00/doc/async_...
[1]: https://news.ycombinator.com/threads?id=seliopou&next=980930...
It takes an interesting, slightly different approach, in my view especially for slightly lower-level operations compared to the "scipipe" pattern above. For example, it looks like something I'd use for implementing the internals of components in my pattern.
What do you mean with "pushback"? Is it the same as "back-pressure"?
(If so, Go has "implicit backpressure" in its bounded buffers on the channels ... although I don't think it is changeable at runtime, unless you replace the channel ... but not 100% sure ...)
At best these are copy-pastable patterns.
(0) Give every process in the pipeline could its own input and output type, rather than forcing all processes to use strings.
(1) Enforce, statically, that only "compatible" processes can be composed. For instance, if `Foo` is a process that outputs integers, and `Bar` is another process that takes strings as input, then `Foo` can't feed its output to `Bar`.
However, this is involves some rather tricky type hackery (so-called "type-aligned sequences"). Hardly a good fit for a language with Go's design goals.
Well, I think unfortunately my sketchy code examples maybe did not cover the idea in enough detail.
What I'm doing in practice is to create different struct types for each type of process I use, which all are of a "base interface" called "process" (mainly for the Run() method, so that the pipeline component can call it).
An example of this in action is shown in this code example on the Go playground: http://play.golang.org/p/voUfPGQulf
The typing of channels for each process struct type should now enforce what components can be connected together (you will not be able to assign a "string chan" to a "[]byte chan" field, for example).
Did this answer the question, or do you see further problems here?
In order to prevent further embarrassing myself, I'm going to tell you how I (currently) understand your design, and, if I'm wrong, please tell me: Your "pipeline" isn't actually connecting the processes to each other. You first connect the processes to each other, and only then add them to a "pipeline", whose primary (only?) responsibility is to ensure that each process runs in a different goroutine.
This much doesn't really need parametric polymorphism, let alone generalized algebraic data types. However, calling it "pipeline" is kind of misleading, because the actual "piping" (connecting processes to each other) is handled elsewhere. In fact, using your "pipeline" abstraction, you could perfectly well run a bunch of processes that aren't connected to each other at all.
Is my understanding correct?
Exactly!
> This much doesn't really need parametric polymorphism, let alone generalized algebraic data types. However, calling it "pipeline" is kind of misleading, because the actual "piping"
Ah, yes, didn't think about that, but yeah, but that's a good point! I should consider a better name for this ... Thanks for pointing it out!
Catnaroek said it already: without parametric polymorphism you are forced to either: 1) commit to a single type upfront which limit the utility (string in your example I think), 2) use interface{} and suffer the boxing in every circonstances while not having any static type checking.
I actually like your pattern. I think it makes sense to have a process as a pair of input and output streams.
See also answer to Catnaroek above!
I think it should read: https://play.golang.org/p/vrwEkHy-DR
In the meanwhile, please see this working example on the Go playground: http://play.golang.org/p/voUfPGQulf
func NewProcess() *A-Process {
....In the meanwhile, please see this working example on the Go playground: http://play.golang.org/p/voUfPGQulf