555 karma · joined February 13, 2011
tldr is Temporal is more general-purpose: for reliable programming in general, vs data pipelines. It supports many languages, and combining languages, has features like querying & signaling, and can do very high scale.
CI/CD is a common use case for Temporal—used by HashiCorp, Flightcontrol, Netflix: https://www.youtube.com/watch?v=LliBP7YMGyA
You do need to run it on the same code version. There are different ways deploy code changes. If you use one of our in-built versioning systems, the version is recorded in the workflow history (and you can either keep track of version-sha mapping or use a sha as the version). Otherwise, you can add code that adds the current code version as workflow metadata.
However, if all of the data you need to update is in a single database that supports atomic commits, I'd go with that over sagas.
Step1
Step2
Step1Undo
then this has a 1% chance of needing manual repair (it's okay if step1 fails, but if step1 succeeds and step2 fails, we need to repair):
do Step1
do Step2
and this has a .01% chance (we only repair if Step2 and Step1Undo fails, 1% * 1%):
do Step1
try {
do Step2
do Step1Undo
}If you're asking whether the catch clause in a Temporal Workflow saga is guaranteed to execute, the answer is yes. The way it's able to guarantee this is by persisting each step the code takes and recovering program state if a process crashes, server loses power, etc. For an explanation of how this works, see: https://temporal.io/blog/building-reliable-distributed-syste...
> source data in CSV files; - Capillaries script (JSON file) that defines the workflow and the transforms; - Python code that performs complex calculations (only if needed).
Temporal is more general purpose: source data anywhere, and you write code to define workflows and transforms instead of JSON, and the code can be in Go/Java/Python/JS/TS/.NET
- Creates and updates events/messages for you
- Maintains consistency between the events and timers and the state of the processing flow. More info on why this is important for correctness/resilience: https://youtu.be/t524U9CixZ0
The difference between workflow code and using an event bus is that with the former, the above is done automatically for you, and in the latter, it's done manually, which can be a lot of code to write and get right, and is harder to track/visualize what happened in production and debug. It would also take a lot of events to get an equivalent degree of reliability—the message processor would need to do a single step and then write the next event to the bus. So a 10-line workflow-as-code function would translate to 10 different events in the bus route.
Also, the event bus route doesn't have the new possibilities I listed in the parent comment.
Temporal, while it uses the "workflow" terminology, is a new type of thing. At a basic level, it's "do you want your backend code to run reliably?" If yes, and you're okay with the latency hit of each step getting persisted for you, then the answer is "use Temporal to write your backend code." It's a new programming model that lets you develop at a higher level of abstraction, where you don't have to be concerned about faults in the hardware or network or downstream services/3rd party APIs being temporarily down. Where you no longer have to code retries, timeouts, use task queues, or use a message bus to communicate between services. And oftentimes don't even need to use a database.
> The Visual nature of the flow, ability to visually diagnose suspended / failed executions.
Temporal has a web UI in which you can see which executions are failing, and see on which step they're failing:
That doesn’t take into account the time it’s already been waiting. For when you want to do something on a schedule and be able to edit the schedule in future, there’s a Schedules feature that allows you to periodically start a workflow, like a cron but more flexible. [2] In this case, the workflow code would be simpler, like just ChargeCustomer() and SendEmailNotification().
So it's great for any code that you want to ensure reliably runs, but having methods that can't fail also opens up new possibilities, like you can:
- Write a method that implements a subscription, charging a card and sleeping for 30 days in a loop. The `await Workflow.DelayAsync(TimeSpan.FromDays(30))` is transparently translated into a persisted timer that will continue executing the method when it goes off, and in the meantime doesn't consume resources beyond the timer record in the database.
- Store data in variables instead of a database, because you can trust that the variables will be accurate for the duration of the method execution, and execution spans server restarts!
- Write methods that last indefinitely and model an entity, like a customer, that maintains their loyalty program points in an instance variable. (Workflows can receive RPCs called Signals and Queries for sending data to the method ("User just made a purchase for $30, so please add 300 loyalty points") and getting data out from the method ("What's the user's points total?").
- Write a saga that maintains consistency across services / data stores without manually setting up choreography or orchestration, with a simple try/catch statement. (A workflow method is like automatic orchestration.)
https://temporal.io/blog/building-reliable-distributed-syste...
Scaling takes a lot of expertise, and I'd say almost everyone saves time & money and gains in reliability by using Temporal Cloud over self-hosting.
Starting a workflow is currently ~40ms, and I think we’ll be able to get down to 10ms this year. How long it takes to complete depends on how many persisted steps it takes (and whether it has to wait on an external event). The only steps that are persisted are workflow api calls like sleep(), startChildWorkflow(), or calling code that might fail (ie “Activity”, like a network request).