YousefED

The downside of building this directly on Matrix events is that I don’t think it would work offline, right? (Until p2p matrix gets more common, that is).

My biased suggestion would be to see how far you get with storing the data to be synced in SyncedStore, and connecting Matrix-CRDT and y-indexeddb for local storage.

Thanks! Personally, I’m looking forward to stronger / cleaner (TypeScript) sdks with for example easier auth and e2ee key management. I heard this is on their roadmap for this year, so that’s pretty exciting!

Fair point haha. I’ll also set up a matrix room and bridge it to Discord.

I just set up a Discord @ https://discord.gg/9GxSXCeQyN. Please join!

It’s called TypeCell as that’s the (yet to be announced) project both Matrix-CRDT and SyncedStore are building blocks for.

If you’re interested in tech like this and maybe also “future of programming” (those bret victor things ;) ), knowledge management software (notion, roam, obsidian) - i’m sure you’ll like it!

Yep, thats pretty much correct. With the addition of sending occasional “snapshots” so clients don't need to read the entire room history initially (more about this in the bottom of the readme).

You authenticate against a Matrix server (the “homeserver”). This can be a server that you host yourself, or one that is readily available (matrix.org is the most popular one).

This design is fairly similar to email where you can be @gmail, @hotmail, or @whateveryoulike.com

Good spot. The WebRTC part is currently experimental and might be superceded by a different solution. If that's not the case I'll probably abstract it out indeed!

Thanks for the questions!

- If you'd enable E2EE for the Matrix client that you pass to Matrix-CRDT, key management is covered in the same way that Matrix does it normally. If key sharing with a particular user is broken, then you won't see updates from that user anymore and v.v. Basically, the transport between you and that user is broken. As CRDTs such as Yjs are designed specifically to work Local-first, at the core there is no assumption that all clients should always be connected to each other. Once the clients are able to communicate again, potential conflicts would be "resolved" according to the CRDT design.

- Redacting: yes, basically you need to trust clients not to fiddle with messages (I think this is fair as you trust them to work with you on the same data already)

- UX / communication: good question! Technically would be possible to purge old (deleted data), but I think this is still something we need to explore together while we start to see more mature software built on these technologies.

- Kevin already answered your performance questions. Matrix-CRDT makes an additional optimization so that not the entire history of the Matrix room needs to be retrieved (see "Snapshots" in the readme)

Exactly. One of the reasons to build on top of Matrix was that it opens the path to end-to-end-encryption. Matrix E2EE (Olm) is heavily based on Signal, and wouldn't be trivial to build from the ground up.

I have a prototype running that uses Matrix-CRDT and Matrix E2EE, and it worked great. It's still a bit of a hassle to set up though (mainly, configuring the matrix sdk correctly), I hope to make that easier later this year.

Thanks! And definitely agree, can 100% recommend TipTap!

Indeed, I think Matrix state resolution has some CRDT-like properties. Afaik, the goal of this is mainly to manage room state across servers when using federation - but I'd have to dive deeper into this.

However, not all CRDTs are the same (actually, I think the definition is still somewhat vague - most things "eventually consistent" could be called a CRDT). Yjs is specifically designed for high-performance operations and has great support for rich-text collaboration. It also works great while offline - and you can connect different providers as you like (for example, store updates both locally in IndexedDB, and transmit messages over websocket / webrtc / and now, also Matrix). Definitely wanted to keep this flexibility.

I’m working on this. One of the projects part if this journey is https://www.syncedstore.org which was recently on HN (https://news.ycombinator.com/item?id=29483913). Shoot me a message to collaborate on this or similar stuff!

Lot's of fun! Thanks for sharing

It doesn't do any diffing, but keeps track of operations underneath, and syncs those. The gist is something like this: - if you push a string "hello" to an array, it will sync an update [push "hello" to Array x] along with some metadata - if you modify a property "prop" and set it to "value" it will sync an update [set "prop" to "value"] along with some metadata

It will not automatically diff these with the previous value (in a distributed system it's difficult to say what the "previous value" was, but it will sync only those properties that have been changed). For text heavy operations, you can use the Text structure which will sync fine-grained operations ([delete 4 characters at position 2] or [insert "world" at position 5]). This makes it suitable for collaborating on rich text documents (similar to Google Docs), see: https://syncedstore.org/docs/advanced/richtext.

This might also be an interesting resource to learn how the Yjs internals work (SyncedStore builds on top of this): https://github.com/yjs/yjs/blob/main/INTERNALS.md

Hi! Great questions :)

First of all, SyncedStore does not implement any CRDT algorithms. Credits for this go to Yjs [1] (and its author Kevin), which it uses as underlying CRDT.

Yjs and Automerge are (afaik) the two most commonly used CRDT implementations. Both have their pros and cons, but Yjs has focused a lot on performance [2].

Automerge has a bit friendlier "Immer style" [3] API. I'm not too familiar with @localfirst/state, but it seems to add a Redux style API on top of Automerge.

My approach with SyncedStore was really to provide an API on top of Yjs that's as simple as possible to use in React / Vue / Svelte or plain JS app. I.e.: only use a single React Hook to observe changes, and use regular Javascript assigments to update values. The API is inspired mostly by Reactive Programming libraries such as MobX [4] (from the same author as Immer).

Hope you're still following along :) Maybe it helps to compare the TODO-MVC applications, as both SyncedStore (https://github.com/YousefED/SyncedStore/tree/main/examples) and @localfirst/state (https://github.com/local-first-web/state/tree/main/examples/...) have implemented these as examples!

[1]: https://github.com/yjs/yjs [2]: https://github.com/dmonad/crdt-benchmarks [3]: https://github.com/immerjs/immer [4]: https://mobx.js.org/

Thanks! Love the excitement :) SyncedStore supports the same backends as Yjs, here’s an overview: https://syncedstore.org/docs/sync-providers.

You can indeed store updates in a centralized database (wrapped with an API to handle auth / accounts) - this is probably the most common scenario.

But you can also go fully decentralized and use y-indexeddb to store updates in in the browser, and use a different transport to sync updates between users (e.g. webrtc).

—-

The cool thing is that the data model behind CRDTs is very flexible in this regard. All changes are captured in small “updates”, and as long as these updates at some point arrive at your device, your document can be updated with the changes (it’s eventually consistent).

The site runs on github pages without a backend (examples use y-webrtc).

Thanks! As it builds on Yjs (many kudos to Kevin Jahns for creating that awesome project + community), any backend that's compatible with Yjs works.

The demos work based on WebRTC / BroadcastChannel (for the same browser / device), but I've listed different backends you can use here: https://syncedstore.org/docs/sync-providers

Hi HN!

With CRDTs (Conflict-free Replicated Data Types) becoming more mature and getting more attention lately, I decided to dive in and use it as the basis for a new project. I love the technology, and how it enables me to create Google Docs style collaborative apps, that seamlessly sync data across users / devices but also work offline.

However, I also found that it's not yet trivial to build! This is why I started building a library that makes developing multiplayer apps super-simple, with a very minimum API surface. (basically, you can just modify plain javascript objects and changes are shared across users).

I'm now excited to share SyncedStore - and looking forward to your feedback :)

It's designed specifically for React, Vue, Svelte but also works with plain JS. By using Javascript Proxies, the data store looks like plain javascript objects, except that they'll now sync automatically across devices / users!

This is why I originally started a project to convert ts to json schema; https://github.com/YousefED/typescript-json-schema (and also check out a good alternative https://github.com/vega/ts-json-schema-generator)

I’m exploring the ideas (an easy to use framework to build local-first [1] apps) in my library Reactive-CRDT (https://github.com/yousefed/reactive-crdt). Feedback welcome!

All credit for the underlying tech to YJS, which has been amazing as mentioned by others in this thread.

[1]: https://www.inkandswitch.com/local-first.html

Happy sponsor here. Started exploring the (previously closed) beta a number of weeks ago, and its really a pleasure to work with.

Have worked with Slate and Prosemirror before, and it feels like tiptap2 hits a better balance; easier API and better documentation. All built on top of the proven prosemirror, so if you need to its still possible to dive into the underlying tech. Great job guys!

What about automatically "crowdfunding" open source projects? Idea: would it be possible to design a system where we distribute a membership fee to open source projects used?

Let's say NPM would cost $10 / month per end-user. This would be redistributed to all the packages you use. (Or, similarly, cumulative membership fees will be distributed according to # of projects that re-use your package).

Have there been experiments like this? Any thoughts?

I'm working on something like this, would be nice to connect - find me on Twitter :)

Relive | Javascript frontend / backend / mobile / 3D / dataviz | Rotterdam, The Netherlands | Onsite

Join us in building a product for millions of outdoor enthusiasts (Cyclists, Runners, Hikers, Skiiers, etc)

Tech: Typescript, React Native, Node, WebGL.

http://relive.cc/jobs or yousef at relive.cc

Very cool. Imo one of the big benefits of Typescript is the possibility to better tool interoperability.

In case you need to do the reverse, for example model-first in typescript and want to generate JSONSchema:

Typescript Code to JSONSchema: https://github.com/YousefED/typescript-json-schema - started this long ago but some great contributors are leading it since!

Thanks - looking forward to your feedback!

I thought your comment was hilarious and a good explanation at the same time :)

Cool! Always fun to play around with APIs like these.

I created a Hyperlapse player where you can also search by city: (for example, videos tagged #sanfrancisco http://lapsetube.com/tag/sf)

I can definitely recommend GSAP from my own experience (we're happily using it in a bunch of projects). They've put a lot of attention to performance and the library is very feature rich (with timeline controls, splittext, etc.)

I actually just loaded emails from both the GMail API and an mbox file into ES. Haven't gotten around to do advanced analytics yet, mainly using it to load into Mixpanel / Intercom to track events such as "customer has replied to one of our emails". We then use this info to do funnel analysis in Mixpanel, and exclude users we're currently in touch with via email from intercom onboarding emails.