thruflo

Yup, this is based on work by Neon. Also the Neon model allows you to sync in a database on demand, whilst scaling to zero when you’re not using it.

The previous PG WASM projects from Crunchy and Snaplet have run Postgres in a VM and required a large 32MB download.

This is pure WASM, no VM, 3.7MB download. Plus it has persistent storage.

This event is the 1 year anniversary of the https://lofi.software meet-ups. It's great Discord community to join if you're interested in local-first software development.

This looks really well done. Feels like a Hashicorp product -- does one thing well, abstracts across cloud providers and provides a tight DX.

Kudos!

This page lists current limitations https://electric-sql.com/docs/reference/roadmap

The key features for us on this are:

1. permissions https://electric-sql.com/docs/usage/data-modelling/permissio... which are defined using DDLX rules, authorise data access and can be used to filter data 2. shapes https://electric-sql.com/docs/usage/data-access/shapes which are the main, more expressive way to control what data syncs on and off the local device, including where clauses, joins, include trees, etc.

These are both in development and due soon. From your comment, I think you’ve seen the shapes branch with where clauses and include trees already working, for example.

In the meantime, the shapes API over syncs the full table. This is temporary and obviously suboptimal but it means you can develop today using the shape APIs and still filter data you display using local queries. Then when the proper functionality lands, the sync will become more fine grained and optimal without your app code needing to change.

Hope that makes sense. We’re very much not a full table sync system. Our role is to provide the best possible model for controlling dynamic partial replication (and to maintain integrity across replication boundaries).

I think this is very smart. Instead of a big ML ops stack, do your model stuff where your data is.

For us, the fact it supports Postgres and SQLite out of the box is awesome. It means we can run a model on the server to generate embeddings, sync these to a local SQLite and then run local inference on the same data structure.

Yup, this is correct.

We have had some requests / discussions around adding hooks to the sync service that will support custom logic on the write path (as per https://github.com/electric-sql/electric/discussions/565). This seems like a good idea but they don't exist yet.

Hey, James here from Electric. Congratulations to the PowerSync team :)

As a sibling comment says, PowerSync actually wrote up a comparison here https://www.powersync.com/blog/electricsql-vs-powersync

Aside from Electric being open source and PowerSync a proprietary service, the primary difference is in the programming model on the write path.

Electric provides finality of local writes. So once a non-malicious write is accepted locally, it is final and won’t be rejected by the server. This simplifies the programming model and means you don’t have to code for rollbacks.

PowerSync is a server authoritative system. Local writes are tentative and can be rejected at the server. You run an API on the write path and write logic to handle conflicts and rollbacks.

The different approaches come with different trade offs, both operationally and in terms of the programming model.

On the topic, if interesting, we have a list of alternative projects here: https://electric-sql.com/docs/reference/alternatives

One of the great things about the Elixir ecosystem is that, in so many cases, the libraries you’re using make the appropriate use of supervision trees so you don’t have to.

Obviously it depends on what you’re building and you can craft your own supervision structures if you want to. But most of the time you can get the benefits of BEAM resiliency without having to drop down into the details.

Both these assumptions are incorrect.

We maintain consistency and integrity across shape boundaries.

Yup, it's not the most sophisticated of cost calculator, obviously there's a wide variation in cost savings from reducing query and compute load and the reduction is app specific.

edit: we based some of the calculator variables on https://fauna.com/blog/compare-aws-aurora-serverless-v2-arch...

Interesting, I don’t know enough about MobX to be sure but yes, you can use the notifier to subscribe to data change notifications and query to get the changed rows.

We’d definitely be interested in chatting through the reactivity model you use if you’d be up for it.

Potentially, but not for a while at least.

Technically it’s possible to support different backend databases. But we want to focus on getting Postgres right first.

Fixed!

Hmm, yes, seems like we might need to bust a cache key there :)

Hey, yup, a big part of the new release is a proper system for dynamic partial replication, as per https://electric-sql.com/docs/usage/data-access/shapes

Hey, cool and yup, that's exactly the kind of setup we aim to support and we have drivers for both Expo and React Native.

Re: RLS, not quite, you'll need to port your rules to our DDLX syntax https://electric-sql.com/docs/api/ddlx -- which is not fully implemented yet (we're working on right now, will be available shortly).

Hey, thanks :) and yup, lots of complexity and engineering. We're lucky because we're building on a lot of research and previous work.

The model is Transactional Causal+ Consistency (TCC+). This has been formally proven to be the strongest possible consistency mode for an AP database system.

We have a page on literature here https://electric-sql.com/docs/reference/literature, which includes the Cure paper http://ieeexplore.ieee.org/document/7536539/ on TCC+ and the highly available transactions paper https://dl.acm.org/doi/10.14778/2732232.2732237

Annette Bieniusa, Nuno Preguiça and Marc Shapiro from our team are authors of the Cure work (amongst many other things!).

Re: rollbacks, our model is to provide finality of local-writes: https://electric-sql.com/docs/reference/architecture#local-w... -- which means you don't have to write rollback handlers.

We cover this a bit in the architecture page: https://electric-sql.com/docs/reference/architecture

Basically Electric is horizontally scalable. You can run multiple Electric sync services on top of Postgres and then obviously many clients connecting to the Electric. We're focused on running on top of a single Postgres right now (which we take query load off).

Hey, very cool :)

We don't recommend production use right now. There's a roadmap page here https://electric-sql.com/docs/reference/roadmap -- we particularly need to flesh out shapes, permissions and validation.

Happy to chat (e.g. on our Discord) if you'd like a bit more detail / looking at whether Electric could fit into your tech pans.

Hey, yup it's definitely a medium term objective.

The team at SkillDevs are maintaining a Daft/Flutter client at https://github.com/SkillDevs/electric_dart

Plus we have a thread to extract the core client-side replication component to Rust to be able to compile for multiple targets.

Hey,

This is the exact pattern we target :)

Drop Electric onto an existing Postgres data model and use the Shape-based sync to manage dynamic partial replication of subsets of data between central Postgres and local SQLites: https://electric-sql.com/docs/usage/data-access/shapes

James

Hey, thanks :)

We have a Discord if you fancy chatting at more length! https://discord.electric-sql.com

Re: CRDTs and intuition, yup, there's trade offs. We are working hard to deliver a solid model to code against. So we have finality of local writes and standard relational integrity guarantees. You can read a bit more about some of techniques here https://electric-sql.com/blog/2022/05/03/introducing-rich-cr...

Re: electrify yes, we create a shadow table and some operation log tables etc. So there is some storage amplification but it's of the order of 2x (we use operational based CRDTs and we garbage collect the operation log). We see the fit being with standard OLTP workloads, mainstream relational apps. We don't target high volume data ingestion, etc.

Re: commercial model, we're designed for open source self-host. We see Electric as a drop in sync layer, not as a full stack backend-as-a-service. Lots of options but concentrating on product for now.

Thanks :) and yup, definitely, we were digging into the typed-sql repo just the other day.

Yup, this is what's addressed by write permissions. You can express who can set salaries and column level rules to validate input values.

When it comes to concurrency problems like not spending money twice, the plan is https://electric-sql.com/blog/2022/05/03/introducing-rich-cr...

With wa-sqlite in IndexedDB mode we're main thread. In OPFS mode it's a worker thread. With the mobile drivers you go over a native bridge.

With Electron / Tauri it's the same as the browser. You're running in Chromium / native WebView on the front-end side.

Hey,

You can see our database rules spec here: https://electric-sql.com/docs/api/ddlx

We haven't implemented it all yet but you can see the intention / direction. It's similar to RLS but adapted for the context.

Connections are authenticated by signed JWT: https://electric-sql.com/docs/usage/auth

We also auto-generate a type-safe data access client from the electrified subset of the Postgres schema. This applies type-level write validation and will apply general write validation when we get there.

James.

Hey, thank you :)

We currently use IndexedDB for the virtual filesystem underneath the wa-sqlite [1] WASM SQLite driver in the browser. But yup, that means a ~1.1MB WASM download.

We'll definitely stay with SQLite in the client. One of our key principles is full SQL support (any Postgres supported SQL on the server, any SQLite supported SQL on the client). So we won't go pure IndexedDB without a database on top.

There's a lot of cool stuff going on with OPFS and WASM SQLite atm, so the technical options are evolving fast.

[1] https://github.com/rhashimoto/wa-sqlite

Hey,

James here, one of the co-founders of Electric.

Both projects are doing active-active CRDT-based sync. Our focus is on sync via Postgres and on compatibility with existing Postgres-backed applications. So you can drop Electric onto an existing Postgres-backed system and it works with your existing data model.

There's also quite a lot of difference in the development model, how we handle migrations, shape-based partial replication, etc. And we're not focused on p2p sync -- for us, everything goes through Postgres.

Hope that helps -- you can read a bit more about the system design here: https://electric-sql.com/docs/reference/architecture