Hacker News Reader

Comment by andy_ppp | Hacker News Reader

Making mistakes means the author delivered a lot of features to users that they wouldn’t have otherwise.

RethinkDB sounds terrifying though, I’d hate for my DB errors in production to literally have no idea what to do, it makes me feel sick thinking about :-)

I think we should aim for kludgy but boring working solutions first - for example I’m tempted to write a redux middleware that sends dispatched actions to the server and manually sync, order and manage conflicts in these events to build a “perfect” system where front end state management and backend CQRS perfectly marry up. It’s a terrible idea because I know how to build the Podcast app I’m making perfectly well without any of this nonsense.

View on HN

For me, the fact that "They allow you to listen to live updates on (almost) any query." would've been a huge yellow (red?) flag. If your database tech does this_amazing_thing_that_nobody_else_does, you'd better be goddamn sure you understand the reason why nobody else does it, AND understand what are the compromises that this brand_new_tech did in order to support that. You probably pay somehow for that amazing feature - what is the cost, really?

WRT "… If your database tech does this_amazing_thing_that_nobody_else_does, you'd better be goddamn sure you understand the reason why nobody else does it …"

I think several databases actually do this.

For example, Alerting in MarkLogic

https://docs.marklogic.com/guide/search-dev/alerts#id_29203

Another example is

Percolations in Elastic Search

https://www.elastic.co/guide/en/elasticsearch/reference/curr...

---

My personal view is that those are useful features, however, often they indicate poor architecture where developers did not create proper APIs that put data into the database, and instead rely on database 'triggers' (or percolations/alerts) to notify the 'world' about changes.

Those kinds of notifications, should really be coming from either data consumer or data producer APIs, -- not from the database engines.

You can do that with PostgreSQL logical decoding, which listens to the replication stream. You can write a plugin in C to emit the updates in whatever form you need.

Right; but you have to implement it yourself, and you'll know the limitations (e.g. it'll probably be expensive to notify on changes to complex queries and/or on many queries).

I actually implemented this (i.e, your parent's idea) for a product. Took me an afternoon to read the documentation and source of the logical output plugin and build a logical replication receiver (using the guts of an existing binary-protocol driver).

Of course, that is a testament to the quality of the documentation and the source code of PostgreSQL.

The fact that the Logical Decoding and Logical Output features were built for replication means that some decisions were taken not-in-favour of publishing changes, obviously, but it works pleasantly and straightforwardly. Even writing a custom logical output plugin would be simple, because Pg does all the transaction tracking, position tracking, change decoding etc.

Why do you need to do that...

https://www.postgresql.org/docs/9.0/sql-notify.html

Should be okay for 99% of needs no?

That sounds like you'd need to manually enumerate all possible queries you'd want to listen to and have all writers do the work of deciding which channels to update.

The idea of simply being able to listen to a query without any additional work sounds like it could be really attractive in certain situations.

"The idea of simply being able to listen to a query without any additional work sounds like it could be really attractive in certain situations."

Sounds a lot like RethinkDB changefeeds https://rethinkdb.com/docs/changefeeds/ maybe check that out first :^)

> I think we should aim for kludgy but boring working solutions first

Nice idea in theory. Can be awful in practice.

For example in enterprises it's very common to see applications last years or even decades. And so that kludgy but boring solution would be the foundation on which you will have hundreds of engineers and years of work built on top of. And because your kludgy but boring solution worked in the beginning you rarely go back and redo it.

Hence why you see a lot of upfront developers built these over-architected monstrosities. It's because they are trying to cater for a decade of possible new features.

This doesn't sound as bad a situation as you make it out to be...

Dev Person: "Hey boss I finished that X, but I used a framework that hasn't been cool for at least 5 years but it will probably last a decade without a rewrite and everyone can build on top of it."

CEO Person: "That sounds great, please keep doing more of that!"

Admittedly, I like playing with the new shinies as much as the next dev, the above scenario sounds horrible to me to :)

You either die a startup or live long enough for everyone to hate the madman who picked the stack.

In my case, you either die a startup or end up having to rewrite the whole thing in React... all 30 years of technical debt.

This is always the best because you get to reimplement misfeatures in your shiny new codebase because “we don’t want to change how customers are used to working”

Bug for bug rewrites are best because “You never know what system out there is relying on current behavior”

I think boring could also be described as easily understandable. Kludgy could be interpreted to mean that all possible theoretical scenarios are not handled. Working could mean that the product works for the use cases it was intended for. In that case, I see this as a very positive pragmatic approach.

You don’t waste time implementing features you “know” you will need in the future (you don’t know). You don’t waste time handling scenarios that make your feature theoretically complete and elegant but that have no significance to the users of the product. You go quicker to market and can more quickly iterate on your product.

I think a much better approach is to implement these kludgy solutions that apply to the specific problem being solved. Then, you can revisit the code when it makes sense (e.g. the scope of the feature is being expanded, or just do it regularly to keep things healthy). Generally holding off on implementing things you don’t need will put you in a better position to implement the things you will need when you reach that point in the future.

Think about the use-case where you want all your Relational data to be denormalised and written to a S3 bucket and then loaded into something like Redshift for analytical loads.

If you start pushing the data from your application itself then you introduce the problem of dual-writes (what if S3 write failed? what if RDBMS failed?).

If instead you treat your RDBMS as the source of truth and use something like PostgresSQL's logical decoding feature to listen to the replication stream and then push the events from that stream to S3 you can achieve guaranteed-eventual-consistency.

The best way would've obviously been to make your application write to a Message Bus (Kafka etc.) and then some other application would persist that message to your RDBMS. This approach requires more development effort and introduces all the issues that distributed systems face: Read your writes, read follow writes, write follow reads etc.

Yes, I’ve worked on systems that use Kafka to do exactly this...

I'd love to discuss more on this topic with you since I am currently working on implementing a proof of concept for the same. Since I've just had almost 1 year of software development experience (2017 grad here) the task feels too daunting to me. Also it's just a one-man team since its a proof of concept.

Oh man, rereading the comments above made me realise i replied to the wrong comment. Doesn't matter though.

> front end state management and backend CQRS perfectly marry up.

I am currently researching how to do this properly. What would you think would be the "boring but reliable" way to do it?

Well I want to do this but the point is I think it's research and unknown; I'm actually saying don't do this! Have an endpoint and call it for changes. No stream of events perfectly queued and ordered between systems. Just accept your app won't be as clever and well designed but it might actually just work :-)

Reply on news.ycombinator.com