302 karma · joined May 1, 2015
https://docs.estuary.dev/concepts/derivations/
Fully managed, UI and config driven. Write SQLite programs using lambdas that are driven by your source events, in order to join data, do streaming transaction processing, and no doubt lots of other things we haven't thought of.
[1]: https://docs.estuary.dev/reference/Connectors/capture-connec...
When passing identifiers, there's generally some combination of lookup tables, per-distribution salted hashes, or encryption happening to make reverse mapping as difficult as possible.
(I was in this space up until a few years ago).
An ability to easily change the boundaries of your conceptual components, because they WILL be wrong now or in the future.
Yes, marketing and docs are not our strongest suits and we need to do better. To be fair, though, we're also trying not to scare off less technical users who see a bullet list like above and think "well this is clearly not for me". It's a hard balance
* Scaling well to multi-TB DBs without pinning the write-ahead log (potentially filling your DB's disk) while the backfill is happening. Instead, our connector constantly reads the WAL and works well in setups like Supabase that have very restrictive WAL sizes (1GB iirc).
* Incremental fault-tolerant backfills that can be stopped and resumed at will.
* Flowing TOAST columns through seamlessly to your materialized destination, without requiring that you resort to REPLICA IDENTITY FULL.
* Being able to offer "precise" captures which are logically consistent in terms of the sequence of create/update/delete events.
The last one becomes really interesting when paired with REPLICA IDENTITY FULL because you can feed the resulting before/after states into an incremental computation (perhaps differential dataflow) for streaming updates of a query.
Our work is based off of the Netflix DBLog paper, which we took and ran with.
[1] https://github.com/estuary/connectors/tree/main/source-postg...
By default, if you then go on to materialize your collections somewhere else (like Snowflake), you get synchronized tables that follow your source DB as they update.
But! You can also transform or materialize the complete history of your tables for auditing purposes from that same underlying data-lake, without going back to your source DB for another capture / WAL reader.
Is the desire for "RT-based datalake" itself misplaced? Or just that the implementation isn't up to the job? Nobody _wants_ slow data, and reports that are usually fine with T+1 delay can become time critical (for example, a "what's selling?" report on black friday).
The former can be faster, because you’re able to walk the table in physical storage order, but your DB cannot reclaim WAL segments while this happens (disk can fill).
The latter allows for resumable backfills that can be tuned to the available spare capacity of the source DB, and doesn’t cause WAL segments to back up, but must walk tables in logical order which can sometimes be slower (for example, if you key using a random UUID).
Otherwise you can’t properly abort the txn because a “select x where y” that was previously empty, no longer is.
Does Debezium's DDD-3 watermark (DBLog) implementation for Postgres not process the WAL quickly enough? We don't use it ourselves either, but architecturally it appears it would reasonably bound how long the WAL can remain un-read?
Agreed that many production DBs people care about have pretty severe limitations here! Managed Supabase is another good example.
(Competitive product and I feel weird about replying in Artie's thread, but tang8330 has said they're not serving this segment)
We put quite a bit of effort into our CDC connectors, as it's a core competency. We have numerous customers using them at scale successfully, but they can be a bit nuanced to get configured. We're constantly trying to make our onboarding experience more intuitive and seamless... it's a hard problem.
You can build cyclic data-flows [2] - not just DAGs - which seems a key ingredient.
You can also write contract-based tests that verify the end-to-end behaviors of your "agents" [3].
Today, you're limited to SQLite or TypeScript. Looking forward, we aim to open it up to any program packaged in a docker container.
[1] https://docs.estuary.dev/concepts/derivations/
[2] https://docs.estuary.dev/concepts/derivations/#approving-tra...
https://github.com/estuary/flow/blob/master/supabase/migrati...
we use macaddr8 instead of bigint, because it has a postgres serialization / JSON encoding which lossless-ly round-trips with browsers and it works well with PostgREST. The same CANNOT be said for bigint, which is a huge footgun.
Confluent calls it a "customer 360" problem [1] and I don't disagree.
We (Estuary) also wrote up a post showing an approach for Slack => ChatGPT => Google Sheets [2], and have more content coming for Salesforce, HubSpot, and some others.
[1] https://www.confluent.io/blog/chatgpt-and-streaming-data-for...
One thing I’ve noticed, though, is that my difficultly in getting into a flow state is more often than not because of a subconscious hang-up about the approach that I haven’t articulated yet.
I’ve learned to recognize and lean into this, and actively look for what’s bothering me about a design. Usually while doing other things.
I suspect it’s actually saved effort overall.
We use RLS extensively with PostgREST implementing much of our API. It _absolutely_ uses WHERE clauses and those are evaluated / indexes consulted before RLS is applied. Anything else would be madness.
Funny. In my experience, application-level authorization checks are very error-prone, easy to accidentally omit, and difficult to audit for correctness."Unmaintainable", I suppose.
Whereas RLS gives you an understandable authorization policy with a baseline assurance that you're not accidentally leaking records you shouldn't be.
Basically you get a provisioned SQLite DB to which you apply whatever migrations you wish, and write SQL lambdas that are run with each input document, where your lambdas update your tables an/or publish outputs via SELECT.
In this world, a separate control plane handling time- and job-based polling has some architectural friction with a data-plane that wants to process records the instant they're available.
Disclosure: I have skin in the game / am co-founder of https://www.estuary.dev/