Turning the database inside-out with Apache Samza (2015)
confluent.io
confluent.io
Service -> Messaging Broker (Kafka) -> Consumers (Samza) -> View, where writes go to service and reads read from a view.
However, I more often see a CDC (change data capture) approach used in practice:
Service -> DB -> CDC -> Messaging Broker -> Consumers -> View
My understanding is that there were three reasons that this was the case:
1) Many already had some conventional DB in place, so it was easier to slap a CDC + Messaging Broker to get a lot of the benefits with less infra changes.
2) You require strong consistency
3) Messaging broker of choice might not be durable enough to entrust your writes to.
My understanding is that Kafka is durable (as this article states), so aside from (1) and (2) is there a strong reason to prefer a CDC approach?
Either way you need to pick somewhere to accept transactions as the ground truth / root store. That root store is the arbiter of which transactions get accepted and which are rejected due to conflicts. (With Kafka you can figure this out through the event stream's order).
With Samza the root store is Kafka, and with CDC its your database.
Its obviously easy to scale everything "downstream" of your root source of truth because everything there can easily be redeployed and trivially scaled for redundancy. But your root store needs to worry about partition tolerance. That said, PostgresQL and Kafka probably scale fine for the needs of 99% of users.
You can do queries either way. Using CDC you can run normal queries directly from your database, or another database downstream if you want to distribute the load. Using Samza you can run a query from any of your views.
The other potential downside I can see is if lost your views for any reason. If your views are not tolerant to partitions, then you face a long time to rebuild it the further out you go, so you may have to snapshot your views at certain intervals and sync with the specific record of the last kafka insert during snapshot, otherwise you have the same problem that people have when they clone a huge long-running git repo from the beginning. (edit: Kafka has log compaction so it may help alleviate some of the rebuilding process)
I've seen jobs spend 6+ hours rebuilding from Kafka when the local store is blown away for all partitions (using log compaction) -- if it's just a partition or 10 (single node failure), it takes way less time.
When it comes to production, Samza's host affinity support is probably the most important thing to have working for when a job fails.
https://samza.apache.org/learn/documentation/latest/yarn/yar...
Give them a google, they are also called Convergent Replicated Data Types, you'll find them very enlightening. We have them as the base conflict-resolution algorithm for our database, http://gun.js.org/ and they work great.
[1] plus, the "right to be forgotten"
Warning: a lot of the literature and talks on ES/CQRS get bogged down in Object Oriented and Enterprise minutia, and detract from the promising core ideas. My advice is to try and see the forest from the trees when Googling these topics.
This was also a inspiration to Redux (React).
For example, let's say I want to model a filesystem on top of the database. And I want to give users access to folders when they have access to an ancestor of that folder. How do I express that in SQL?
Also how would revoking access work in the proposed "data-stream" model?
Is this a dig on those of us who have not gone to pure FP languages, or just those who use globally scoped (unscoped) variables?
Aka marketing
Fine but the semantic of your events will be exceptionally important as will versioning if you'd like a consistent way of making your views re-buildable. This is easy to mess up.
Kafka + YARN + Hadoop + DBs + code. To me this is neither elegant nor simple but perhaps necessary complexity if you're into something in realtime and on scale.
It doesn't look simple from the outside, but thats how complex any modern database is. The only difference with samza is that you can decide how much of that complexity to buy into. And because you have access to the "internals" of how data flows through the system, you can extend that view through your application. (Eg by making a standalone access control filter which only allows some users access to some documents. Or wrapping your react static HTML rendering code as a database view that you can, for free get a change feed to.)
For me the complex part that I can't quite stomach is having to buy into hadoop and zookeeper. But thats a tooling problem, not an architecture problem. We need lighter implementations of all these tools.
Why does it have its own stream processing when there are 5 or so ones already in Apache?
The tl;dr is that Samza is closest to Storm+Trident in delivery guarantees, but is more flexible about how data can be passed around (pluggable serializer/deserializer (serde) architecture), and offers an out-of-the-box solution for state management (RocksDB).
- Kafka writes its topic offsets into, er, Kafka, rather than Zookeeper; which means ZK load is much lighter. I'd be interested in knowing more about why people find ZK heavy weight - I've found it to be light, unobtrusive and very robust. I also use it for managing state (distributed locks) as brokers boot up and obtain a consistent Broker ID from a defined list of available 'slots' (e.g. if broker 2 goes down, I want the replacement node to also boot up and be assigned broker ID 2; if two nodes go down, I want those free slots to be allocated to the next two nodes that boot up).
- Kafka v0.10 released the Streams API, which has many similarities to Samza, but removes the requirement to run another cluster for your streaming applications (no YARN requirement - phew! Kafka will load balance your consumer-group streaming-app; kill/restart/boot as you see fit). Consume data as streams and build virtual tables (persisted to disk via RocksDB for get/put operations, written into Kafka for availability/persistence); join stream->stream, stream->table, filter, map, aggregate via time windows, write your own processor/transform classes.. it's very flexible without always needing to dive into the lower level Processor API.
- Kafka v0.10.[01] also included an Interactive Queries API - what you get when you realise that your stream apps have created intermediate aggregates/tables of information that would be so handy to query (by other apps, or internally, from across the cluster). I haven't used this API yet, but anticipate doing so in 2017.
Samza to me always felt like it was in tech preview - not comprehensively documented and the only real source of knowledge was reading the source and asking questions in the mailing lists (of which there were excellent and high quality answers).
What other architectures are people using for streaming data pipelines/apps (things that do more than count clicks, i.e. transforms, enrichment, etc) - i.e. lowish latency (event to output in less than, say, five seconds) where relational databases aren't a natural solution?
I'm leveraging Kafka at my day job as a distributed data backbone (company-wide focus - think application logs, app usage logs, usage-derived billing, subscriptions, master data, etc), which when coupled with Streams (new breed of application at this company) gives us our first steps towards event-driven processing and data delivery as a stream of new information, rather than periodic batches.
This provides us with an alternative to the API+database model we've used for more than a decade and which works very well, but there are times when polling isn't the answer. Decoupling producers of data from consumers has subtle but substantial benefits (data liberation, for one, removing/reducing point-to-point integrations for another).