Comment by justinsb | Hacker News Reader

Comment by justinsb | Hacker News Reader

The idea of the log goes well beyond just real-time data (as the blog post describes, although the title does not). I think it might well turn out to be one of core building blocks of _all_ stateful systems. Amazon's Kinesis has for the first time exposed a reliable Log-aaS on the cloud; I think we'll start to see more systems built around it.

My personal "24 commits for December" project is to build a set of open-source cloud data-stores, all backed by a distributed log using Raft, "blogging all the way". I'm half-way through, and I've implemented a simple key-value store, put a Redis front-end on it, used that to implement a Git server, and am currently working on building a document store with SQL querying. All with the same architecture: the log provides fault-tolerance and consistency, we have a data structure specific to the particular service (e.g. message queue or key-value store), we periodically take state snapshots so that we don't have to reply the whole log after every failure.

Feel free to follow along / provide feedback: http://blog.justinsb.com/blog/categories/cloudata/

Yeah totally agree. I focused on real-time data because offline data processing is often able to be less principled about the usage of time and doesn't need an explicit log (e.g. many ETL pipelines work this way).

Hi there - I see you're Mr Kafka :-) I wanted to use Kafka instead of Raft for my project, but for my application I couldn't tolerate the (even highly unlikely) possibility of losing a message (when we lose all the nodes in the In-Sync Replica set). I understand why this tradeoff is there (for the clickstream use-case), but I hope it might be possible to retrofit reliable writes to support other use cases as well. The more open-source reliable log services we have the better!

Yup, agreed. Wanna help? :-)

I'd love to help (I opened KAFKA-1050 on the issue), and I even started coding, but I got stuck on how to deal with the rollback problem when a write isn't acknowledged by a quorum. I think it requires a "real" distributed consensus protocol (unlike the current design), which is a fairly big change! It may be that the answer is to use Kafka's efficient storage design with Raft, which is something I explored a little on day 1 of my project: http://blog.justinsb.com/blog/2013/12/07/cloudata-day-1/

First, great article.

Coming across article now was quite serendipitous as I'm in the process of designing a Databus like system myself. In regards to Kafka and Databus, one thing I've been wondering about is why the Databus relay isn't implemented with Kafka? Kafka seems to provide the same semantics (at-least-once in order delivery, clients can pull from arbitrary positions in the stream, etc.). I know Databus provides transaction semantics in the stream and a few other differences, but those differences don't seem too large. Is it because Databus is to be embedded in Espresso? Or maybe even something as simple as two different teams converging on the same solutions?

Anyway, thanks again for the article.

Reply on news.ycombinator.com