Hacker News Reader

> Logical operator graphs for graph dataflow-style execution (think TensorFlow or PyTorch, but for data frames)

> A multicore schedular for parallel evaluation of operator graphs

Does anything like this already exist somewhere?

Spark SQL does exactly that: declarative graph dataflow computations + a multicore (also distributed) scheduler and executor, although it doesn't really scale down that well to a single machine and the execution layer happens mostly on the JVM, even when using PySpark (with an ungodly mix of of serialized Python and JVM objects and code).

Ibis (also by Wes McKinney) does the first part, but it offloads scheduling and execution to the underlying database you are using.

We are using Airflow for this

We use a luigi type app for this.

Reply on news.ycombinator.com