Hacker News Reader

Comment by iblaine | Hacker News Reader

Workflow engine = can support hundreds of parallel workflows. This includes airflow, luigi, dagster, appworx, are used to manage data and are typically processes that run in minutes to hours. Orchestration engine = can support millions of parallel workflows. This includes Uber Cadence & Netflix Conductor, are used to manage services and are typically processes that run in microseconds to minutes.

View on HN

Cadence does support processes that run unlimited time. We have workflows in production that are always running. For example there are services at Uber that have an always open Cadence workflow per rider.

Uber could've used Cadence in place of Piper/Airflow isn't it. Can you shed more light on where both of these fit.

Very interesting. Can you give an example where you'd want to track a process that takes days or months to execute?

For example the Uber loyalty program needs to accumulate points (similar to airline points). A customer workflow receives trip completion events and updates the state accordingly. When a certain number of points is reached some actions (mostly calls to downstream services) are executed.

For this use case, could it be solved with an asynchronous task system such as Celery? What advantages does Cadence offer for something like this, which seems to be repurposing the scheduling system of Cadence to process instantaneous events?

Also how does it listen for events? From polling?

It could be solved by Celery, but it would also require a database and Celery doesn't scale that well unless runs on top of Redis which is not really fault tolerant. Also actions invocation with guarantees and exponential retries is not trivial with Celery.

Actually at Uber large number of services are being migrated from Python/Celery to Go/Cadence.

Reply on news.ycombinator.com