Hacker News Reader

Comment by foolfoolz | Hacker News Reader

> Standby Event Store: To avoid loss of messages in the event the MSK is unavailable, the platform is wired with a standby data store, where it writes events onto a database . The architecture provides tools and utilities to read messages and publish them back onto MSK, once it’s available.

on one hand aws msk is good enough for an enormous application like mcdonald’s. on the other they need a backup database just to get around it not being available? what’s the real story here. interested to see where this goes

View on HN

Probably nothing more than a requirement for resiliency. With thousands of restaurants, chances of a couple of them losing connectivity each day are going to be pretty good, through traditional interruptions like backhoes digging up cables and drunk drivers taking out telephone poles.

If those messages are discarded because the store can't talk to MSK (or MSK is unavailable), then things like automatic replenishment based on order volume couldn't happen. The store manager would have to do a daily physical inventory count to know how many bags of fries, boxes of drink straws, etc. to reorder.

But it's more likely that they'll lose connectivity to AWS altogether rather than MSK going down itself. I'd not try to write from each store network directly, but through some gateway that would take care of writing to Kafka and that separate store if we still have availability issues.

Cloud services do go down and it's out of your control when they do. If something must work, you need redundancy.

Kafka is often used for financial applications that must not miss events, so having a backup buffer is a reasonable strategy for those use cases. Things like tracking data is likely not worth backing up due to high data volume and low external visibility when data is dropped.

We use kinesis a lot at work, and some services are architected to write events to a Postgres table which eventually gets dumped to kinesis, while other services write directly to kinesis.

Guess which services fared better during the last kinesis outage?

What's the point of using kinesis in this case? Why can you not consume from postgres. My understanding would be that you use kinesis for impedance matching on writes. If you can already write everything on Postgres reliably I cannot see any immediate use case for Kinesis. I could see the other way around, i.e. having kinesis as a resilience backup for postgres being a more valid concern.

I'm not the person you're replying to, but I've used a similar pattern, and there are a couple of reasons.

First, if you have lots of databases and other applications, then you are talking about a mesh of event busses - which defeats the purpose. Pushing the messages out of the various databases and into a central message bus makes the messages more easily consumable without having to know where they come from.

Second, by writing messages to a PG table first, those messages become part of the update transaction. This means you can post messages at any time during your business logic processing, but if you hit an error and roll back, those messages (which would, presumably, no longer be valid) are also rolled back.

Combine this with message idempotence and you get a very reliable messaging environment.

Reply on news.ycombinator.com