Hacker News Reader

Comment by candiddevmike | Hacker News Reader

Where's the hitchhikers guide to moving back to relational data after our whiz bang dev made us event driven and left for a new opportunity?

View on HN

Next to the 97 part video tutorial on how to get back from your semi-hydrated micro-frontend to something sane for your 12 person company ;)

THe problem is everyone's interpretation and implementation of micro-frontend is different.

So long, and thanks for all the eventual consistency issues

Actually event driven stuff is highly consistent. Event insertion is fast as hell and all of it is timestamped and tagged with a unique Id so there's no consistency issues.

The problem is availability. You have to aggregate those events to get anything meaningful about it and aggregation can be really really slow.

Exactly. Because the aggregation is so slow, you cache the aggregates, but then the caching layer becomes slow to invalidate, and you get eventual consistency problems.

I wouldn't call that a consistency issue. That's just lag. The aggregation is valid for a specific time point. Caches aren't the source of truth. The source of truth remains consistent here.

It's hardly a cache if it's the only way to query the data, it's a materialized view

Then it's an outdated materialized view. If you don't refresh your view from a browser is it a consistency issue? No. Not in in the way the term is usually used.

Your using terminology in non standard ways.

Read the first paragraph on eventual consistency on wikipedia, it describes the agreed upon definition.

"Eventual consistency is a consistency model used in distributed computing to achieve high availability that informally guarantees that, if no new updates are made to a given data item, eventually all accesses to that item will return the last updated value"

False. And wrong.

Read: https://en.m.wikipedia.org/wiki/Consistency_(database_system...

We are talking about the word consistency. Which is most often used in the context of databases. Eventual consistency is here:

https://en.m.wikipedia.org/wiki/Eventual_consistency

It refers to the same concept and technically and colloquially its referring to databases or things similar to databases. If it didn't then every system on the face of the earth is eventually consistent because of browsers and caches and timing. Every browser eventually presents an outdated view if it's not refreshed. If that's the case what is the point of the word? The word is obviously used for categorization.

Thus The scope is usually and colloquially a database systems or some combination of services that represent a source of truth.

If you read further into the article you cited you will encounter this:

"In order to ensure replica convergence, a system must reconcile differences between multiple copies of distributed data. This consists of two part..."

Literally the article assumes we are talking about distributed systems where replicas can exist.

Is a cache a replica? Is a browser a replica? No. The scope is obviously the source of truth at the resolution where you can have replicas aka: two or more sources of truth.

The event store and the cache are both part of the same system, and it's this whole system that is "eventually consistent".

Say I create widgets on screen 1 and they are persisted with event sourcing into Postgres. I see the list of created widgets on screen 2, loaded from a materialised view in Postgres (or Elasticsearch). The "lag" between it being created on screen 1 and appearing on screen 2 is the "eventual consistency" issue I'm referring to here, whereas I think you're referring to the consistency only of the persistence on screen 1.

I'm sure we both agree there's no getting away from CAP theorem. Event sourcing accepts less consistency, and every part of the system needs to deal with that.

>The event store and the cache are both part of the same system, and it's this whole system that is "eventually consistent".

Then every single system of the face of the earth at a high enough level has a consistency issue. Just go to the level of the full client and backend system using web browsers. You don't refresh the browser you of course will eventually have a "consistency" issue.

Usually when they refer to this stuff it's referring to the source of truth: The database. When you shard the database into two synchronizing copies, you increase the availability with a second copy but that leads to the potential for both copies to be inconsistent.

Why not get rid of git too while you're at it?

Just store the current state of the code base. You can't have merge conflicts without conflicting commits.

Exactly. These kind of devs (like OP) cannot be quiet and need to constantly introduce something “cool” so that they can get a good salary raise. The moment they cannot introduce more BS, they move on to another company. The poor other “average” devs need to maintain all the crap.

Do it like an accountant would:

Throw away all invoices and receipts, and just represent each customer's balance as a single number. When it changes, get your crayon, cross it out, and write the new number.

He tried at the beginning to say only when the DB gets too big should you even think about this but there are things you can do there as well.

The way I see it, either your business domain requires querying over a large amount of data, or it doesn't.

If an application allows someone to be able to enter let's say an order number from anywhere in the world from the last 10 years and be able to find the order, there is no magic - some server out there is going to have to scan a huge amount of data to find a match.

Tricks such as indexes, partitioned tables, etc can be employed, but those tricks have nothing to do with event-sourcing and are independent of it.

> Tricks such as indexes, partitioned tables, etc can be employed, but those tricks have nothing to do with event-sourcing and are independent of it.

You might want to use different tricks in different situations. Different situations means different services, and different tricks means different storage/query technologies.

So how do you get your data into three systems - and more crucially - keep them in sync? Webhooks? Triggers? Some bidirectional sync magic app that claims to beat CAP?

Just use event-sourcing (append-only, disallow modification) and the multiple systems will stay in sync as long as they know how to process one more message.

Agreed with your points but this article seems to present event-sourcing as a replacement for your database(s) and even makes claims about saving storage space, thus at least hinting at not using databases anymore.

It's a replacement for your source-of-truth, not your database(s). Although you're right about the article not explicitly mentioning slurping the events back into a DB. I suspect the reason is that there are plenty of articles which explain how to event-source from greenfield, but this is the first one I've seen which focuses on existing brownfield relational data - see the title.

> makes claims about saving storage space

I don't think that was the right reading about saving storage space:

> We’ve been trying to optimise the storage size; we’ve made some sins of overriding and losing our precious business data

I think it's his strawman RDBMS developer who optimised for saving storage space, and lost business data as a result. The suggested approach is:

> We can optimise for information quality instead of its size.

There is no “too big” in databases and in particular size of data is not the criterion for deciding on using something like event sourcing. It’s a really niche paradigm that is only ever going to be useful in quite unusual circumstances. Most of the time people don’t need it, and most of the time when people do it, they find their immutable event source is very inconvenient for lots of the normal things you want to do with your data, so they end up doing something like CQRS[1] (ie having a database as well). This is one of those Martin Fowler[2] type things that looks good on a whiteboard but most people would be better off avoiding most of the time.

[1] https://en.wikipedia.org/wiki/Command_Query_Responsibility_S...

[2] This Martin Fowler, https://en.wikipedia.org/wiki/Martin_Fowler_(software_engine... not this Martin Fowler https://metro.co.uk/2023/11/06/eastenders-star-reveals-why-m...

This reminds me of https://youtu.be/b2F-DItXtZs

Reply on news.ycombinator.com