Go-taskflow: A taskflow-like General-purpose Task-parallel Programming Framework | Hacker News Reader

Go-taskflow: A taskflow-like General-purpose Task-parallel Programming Framework

github.com

76 pointsnoneback22 comments

I have this https://github.com/Azure/go-asyncjob library as well, with generic strongType connecting each step input/output.

CUE has a package (cue/flow) that has a default implementation but also allows you to build custom tasks on top of the engine & evaluator. The nice thing about this setup is that it infers the dependencies between tasks based on the references to inputs/outputs between them. The main drawback of most of these things is that they don't scale beyond a single machine. That's why people choose tools like Airflow.

https://pkg.go.dev/cuelang.org/go@v0.10.1/tools/flow (cue cmd is the default implementation of this)

https://docs.hofstadter.io/getting-started/task-engine/ (custom example)

Agree. lightweight embedded libs like go-taskflow are used in a much different way compared to distributed workflow systems. Coders should be aware of the pros and cons of those repos or systems.

For what is worth I spent couple weeks evaluating Cue and I don’t think it’s mature enough for production use in context of task runner.

There are undocumented quirks (and documentation gaps) requiring hours of experimentation, some subtle bugs and ergonomy paper cuts.

For solution integration solution has to be approachable, and as much as I have sympathy for Cue it just isn’t one. It feels like one needs PhD in software engineering to use it.

This is IMO reason why people go for things like Airflow - popularity and ease of entry. Using Cue in distributed environment isn’t that hard. It can be used directly from Go or wrapped in Erlang/Elixir port.

Shameless plug for a similar but totally different system that I've used to pretty good effect lately: https://github.com/gaffo/jorb meant for just trying to run batches of items with many steps in a simple way.

A major benefit of my tool is that it ends up being pretty easy to re-wire the flows of tasks through the state tree. I tried for a ipynb type workflow as much as possible where I'm storing state on every state change, this lets me kill and restart the processing easily and often only losing the work that was in process on the in flight state changes. I find myself wiring up the first few steps of the state diagram and making the ones I haven't figured out yet temporariliy terminal. Then my computer is crunching away on pre-work and I let one item through at a time in the debugger, figure out that state, and move to the next state, and restart the flow. New items now start processing through the newly non-terminal state. I set breakpoints at all of the exceptional states and let it run working on the next step in the flow. I handle the exceptions and then open the flood gates with parallelism on the state, while also coding up the new state transitions. This lets my computer do a lot of work against things that have concurrency limits or rate limits while working, lowering the totally wall clock time of the large batch job.

What do I use it for for reals? I've been using it to check out and modify large swaths of code bases and build them against a build fleet where I'm rate limited on the build submissions and the ability to pull the code down and modify it. Or where I need to do walking of a large number of expensive apis over time to do complex calculations on small (1-5k sets of data).

I've always thought of building a dag renderer for the graph but I'm changing the graph during execution a lot, what I actually want is to render the state changes of a given task

Yep. Rendering dynamic graphs is complicated, cuz dependencies change all the time.

That's why I decided to support static graphs in the first place. After all, in most cases, static graph programming is enough, and being a user-friendly really matters.

Visualizer and Profiler for graph-based task execution are crucial to improve user experience.

This is a pitfall. I’m working on something similar (internal solution) and dynamic task graph bites back hard very quickly.

Domain I’m working on is very prone to a failure so having simple visualization/interaction (e.g. click to inspect/retry) is an important feature.

For that task graph needs to be resolved and static before actual execution phase. Nothing prevents multi-loop resolution (i.e. task graph which result is a task graph). Just make sure artifact sits in-between.

Totally agree. Dependency handling is quite tricky in a changing Dynamic Graph.

Most credit to taskflow: https://github.com/taskflow/taskflow

A taskflow-like General-purpose Task-parallel Programming Framework with an integrated visualizer and profiler for Go, inspired by taskflow-cpp, with Go's native capabilities and simplicity, suitable for complex dependency management in concurrent tasks.

Is it like Temporal?

Not quite like although they may share some same concepts while using. go-taskflow is more like a lib or an embedded framework to organize task dependency and execute tasks in parallel rather than a platform.

v0.1.2 released. I refactored and simplified APIs. I guess it is more coherent and easier to use than it used to be.

https://pkg.go.dev/github.com/noneback/go-taskflow

Looks interesting, but how do you pass data between tasks?

It just uses native go function closure to hold and pass data among tasks, and it requires users to consider the data race problem. Cuz in some cases, tasks are executed in parallel.

Do you have an example of this done idiomatically? All the examples seem to be just printing, no passing any data between the nodes. That's not really a realistic scenario, since the main point of a flow graph in my experience is to compute stuff, and you typically need back pressure, queues, priorities etc. to solve real-world problems.

Good point. I will add some typically realistic scenario examples.

I added a more realistic example, parallel_merge_sort, which demonstrates how to pass values and how to avoid a data race among tasks.

https://github.com/noneback/go-taskflow/blob/main/examples/p...

Very helpful, thank you. This also makes it apparent (it wasn't clear to me before) that each graph is declared alongside the data. You cannot reuse a graph to run multiple times on different data; you define the graph every time.

Yes. Data is attached to tasks via Go Native variable binding. Values are captured by task function once defined (closure). In some cases, You can change the value of the variable(like changing the value of ptr, rather than ptr itself) and rerun the whole graph to support your cases.

Reply on news.ycombinator.com