Kappa Architecture Is Mainstream Replacing Lambda (2021)
kai-waehner.de
kai-waehner.de
It seems that the field of data science has coined the term “Lambda Architecture” to refer to a common approach to handling data analytics in near-real-time where an incoming data event is sent to two separate systems, the “batch” system and the “live” system. The “batch” system is thorough: it aims for completeness/correctness (every datum has as much analysis extracted as possible, both from it and from its relationship to other data) and consistency (no datum is lost). Since consistency is eventual and correctness is computationally expensive, this system is slow, and it works by collecting a bunch of events and processing them all together in a batch (where it gets its name). So we also send the event to the “live” system, which operates in real time but is not as accurate or detailed.
It is apparently called Lambda Architecture because this two-path setup resembles the lambda character. Because “lambda of x” also means “function of x”, you will sometimes see people try to formalize it with expressions like “λ (Complete data) = λ (live streaming data) * λ (Stored data)”, which is cute, but I don’t think this is really core to understand capital-L Lambda Architecture.
The problem with Lambda Architecture seems to be that the batch system is an operation on (d_0, d_1, … d_n) to produce an output model m, and thus can extract information from things like “d_n has a strong relationship with d_504” - but the live system is just an operation on (m, d_n) and therefore misses that link.
Kappa Architecture seems to just be the idea that you don’t need a batch system if your live system is really good - you can just take the output of the previous live operation and use that as the new m for your next operation. The problem with Kappa architecture that other commenters here are highlighting is that this results in holding a large m in memory at all times (fragile, expensive), and handling out of order events by re-doing the live operation on them - if you do (m, d_505) and then d_504 drops in after, you have to do ((m, d_504), d_505).
There is a Delta Architecture as well, which is just the idea that you don’t need a live system if your batch system is really fast. There’s also a Zeta Architecture, which is just the idea that if you add half a dozen more systems alongside batch and live, nobody will know what’s causing the bugs and you can’t get fired.
This is true for time-sensitive workloads, and most business applications do not have such a requirement and typically have some leeway to tolerate the untimliness of the event arrival. Provided the event has an epoch associated with it that can unambiguously reference the point in space and time when the event had occurred, the time ordered sequence of events can be reconstitued by using tumbling (or another window type) windows (e.g. Kafka and similar) or database window functions. The now time ordered sequence of events can then be flown into a new topic (Kafka) or a DB table/document collection and ingested out of that one in the ordered sequence.
RE: batching and real-time. The only real difference is that real time data is constant vs burst, i.e. a trick flow of data vs a scheduled at regular intervals surge. But a batch can be de-batched and unwound into a series of isolated events at the ingestion point – to become more «real-time» and to not require to be processed as an atomic unit. Therefore, if the untimeliness can be tolerated to a certain degree, there is no real need to treat real-time and batched up data separately.
this would be a fun show. I'd subscribe to Disney plus to follow the unroll of this comedy.