30,161 karma · joined March 17, 2008
Working on sync engines, PGlite, and Tanstack DB at Neon/Databricks. Previously co-founder ElectricSQL (acquired by Databricks).
sam dot willis at gmail dot com
samwillis.co.uk
Twitter: @samwillis
This is a game changer, all the models before that output meshes were a toy at best. Super excited to see where they can take this.
I wander if the next step is for a step -> proprietary format (SolidWork, NX etc) model that can infer constraints.
Did you run any tests with the new transaction system in ClickHouse? It would be super interesting to see how it effected the batch updates.
Please do go and check up what the state of using types to inform the compiler is (I'm not incorrect)
On the area allocator, I wasn't clear enough, as stated elsewhere this was in relation to having something similar to isolates - each having a memory space that's cleaned up on exit.
Python has almost identical semantics to JS, and has threads - there is nothing in the EMCAScript standard that would prevent them.
I suggest you go and read the EMCAScript standard: https://ecma-international.org/publications-and-standards/st...
There is nothing in there about browser APIs, and in fact it explicitly states that the browser runtime, or any other runtime + api are not EMCAScript.
Nowhere did I say that full, or even any, compatibility with Node is needed - it isn't.
We need to stop conflating JS the language with the runtimes.
A JS runtime absolutely can get by without a GC, you just never dealloc and consume indefinitely. That doesn't change any semantics of the language, if a value/object is inaccessible, it's inaccessible...
An arena allocator provides a route to say embedding a js-to-native app in a single threaded web server like Nginx, you don't need to share memory between what in effect become "isolates".
I don't think anything changes with compile to native on the server.
I work at Electric and started the PGlite and now Tanstack DB projects. The issues mentioned with PGlite are one of the major motivating factors behind Tanstack DB. We are taking those learnings and building, what we believe, is the missing client side datastore that is "sync native" and completely backend agnostic. Also being JS, rather than WASM, solves many of the slower than ideal query semantics, and has enabled us to build an incremental query engine for it.
It's also important to note that Electric doesn't require PGlite on the client, far from it - it's essentially a "protocol first" sync engine, you can use it to write into and maintain any client side store.
This solution by the OP, diffing based of modified data is ideal for a huge number of apps, and something that we intend to built into Tanstack DB so you can easily sync with no additional infrastructure.
SQLite (or PGlite) in the browser is awesome, and has the advantage over Tanstack DB at the moment of having persistence (it's on our roadmap), but they are also somewhat chunky downloads. For many local-first apps that's not a problem though.
- Porffor can use typescript types to significantly improve the compilation. It's in many ways more exciting as a TS compiler.
- There's no GC yet, and likely will be a while before it gets any. But you can get very far with no GC, particularly if you are doing something like serving web requests. You can fork a process per request and throw it away each time reclaiming all memory, or have a very simple arena allocator that works at the request level. It would be incredibly performant and not have the overhead of a full GC implementation.
- many of the restrictions that people associate with JS are due to VMs being designed to run untrusted code. If you compile your trusted TS/JS to native you can do many new things, such as use traditional threads, fork, and have proper low level memory access. Separating the concept of TS/JS from the runtime is long overdue.
- using WASM as the IR (intermediate representation) is inspired. It is unlikely that many people would run something compiled with Porffor in a WASM runtime, but the portability it brings is very compelling.
This experiment from Oliver doesn't show that Porffor is ready for production, but it does validate that he is on the right track, and that the ideas he is exploring are correct. That's the imports take away. Give it 12 months and exciting things will be happing.
For TypeScript it uses the types as hints to the compiler, for example it has int types that alias number.
Very early still, but very cool.
There was a really good thread on Twitter a couple of days ago:
> In light of recent Figma news, lemme reiterate that of all the goods that can happen to the web, 90% of them can't happen due to not having access to font rendering & metrics in JS
https://x.com/_chenglou/status/1951481453046538493
And a few choice replies:
> t’s kind of crazy that a platform specifically designed for presenting text doesn’t provide functionality to manipulate text at a detail level
> Brute forcing text measurement in tldraw breaks my heart
Love it or hate it, the web is a platform for application development, making this easer is only good for everyone.
My argument on web APIs is what we should continue to go lower level, and so font and text metrics APIs for canvas would be awesome and an alternative to this. But I'm also a proponent of "using the platform" and for text layout, web engines are incredible, and very performant. Extending that capability to layout inside a canvas enables many awesome features.
One that I've repeatedly gone back to over the years is paginated rich text editing. It's simply impossible to do with contenteditable in a product level way - one of the reasons Google docs has a custom layout engine. This proposal would enable full use of contenteditable for rich text, but with full page/print layout control.
I hope it lands in the browsers.
It's designed for sync, so rather than fetching you can hook it up to a sync engine (any!) to keep your front end in sync with your backend. It's built on Tanstack Query, making the sync engine optional, and a great path for incremental adoption.
The query engine uses a typescript implementation of differential dataflow to enable incremental computation of the live queries - they are very fast to update. This gives you sub ms fine grade reactivity of complex queries (think sql like joins, group by etc).
Having a lot of fun building it!
https://tanstack.com/db/latest https://github.com/TanStack/db
It keeps WASM simple and provides a thin layer as an optimisation.
The issue comes down to the fact that even if your WASM code can return a utf16 buffer, to use it as a string in JS code the engine needs to make a copy at some point. The TextDecoder api does a first good job of making this efficient, ensuring there is just a single copy, but it's still overhead.
Ideally there should be a way to wrap an array buffer with a "String View", offloading the responsibility of ensuring its utf16 to the WASM code, and there being no copy made. But that brings a ton of complexities as strings need to be immutable in js, but the underlying buffer could still be changed.
I don't think that's quite correct. I think the authors fully acknowledge that the business case for local-first is not complexly solved and is a closely related problem. These issues need both a business and technical solution, and the paper proposes a set of characteristics of what a solution could look like.
It's also incorrect to suggest that local-first is an argument for decentralisation - Martin Kleppmann has explicitly stated that he doesn't think decentralised tech solves these issues in a way that could become mass market. He is a proponent of centralised standardised sync engines that enable the ideals of local-first. See his talk from Local-first conf last year: https://youtu.be/NMq0vncHJvU?si=ilsQqIAncq0sBW95
So much truth to this post.
There was a great panel discussion this year from a number of the co-authors of the the paper linked, discussing what is Local-first software in the context of dev tools and what they have learnt since the original paper. It's very much worth watching: https://youtu.be/86NmEerklTs?si=Kodd7kD39337CTbf
The community are very much settling on "Sync" being a component of local first, but applicable so much wider. Along with local first software being a characteristic of end user software, with dev tools - such as sync engines - being an enabling tool but not "local first" in as much themselves.
The full set of talks from the last couple of years are online here: https://youtube.com/@localfirstconf?si=uHHi5Tsy60ewhQTQ
It's an exciting time for the local-first / sync engine community, we've been working on tools that enable realtime collaborative and async collaborative experiences, and now with the onset of AI the market for this is exploring. Every AI app is inherently multi user collaborative with the agents as actors within the system. This requires the tech that the sync engine community has been working on.
(Aside: I think this is something PGlite helps with if your in Postgres land)
Key thing I tried to emphasise what that this whole process was new and that before November last year wasn't possible.
They really got it, my younger son is very excited about the idea of build games that follow the stories he comes up with (he's recently been spending time writing stories on a iPad, inspired by his novelist mother). We're going to spend more time experimenting together over the summer holidays.
Kids are curious sponges, I don't think you need to spell out to them exactly how it works, just show them and their curiosity takes over.
I know at school the teachers have been using image GenAI in English lessons with my older daughter, using it in lessons about descriptive language. They have had the kids experiment with describing things and getting images back. I was quite impressed to hear they were doing that, it's a great way to introduce the concepts in the context of a topic they are covering.
On the general topic of tech we have always (from as soon as they could hold a device) let the kids play computer games, and experiment themselves with tablets. But we've had the internet locked down, and not let them have things like YouTube Kids, it feels too close to social media, and serve explained the dangers of that to them. So very pro exposing children to tech, but no social media at all. I think in time we will try and explain the danger and downsides of AI, but it's all so new, there's not much to cover yet, particularly as we are still developing our own opinions.
People do not want invasive glasses, even if they make them as small at normal glasses. I just don't see it becoming anything other than a niche product.
It's like all the moves to voice/audio interfaces powered by AI. They simply won't take off as audio is inherently low bandwidth and low definition. Our eyes are able to see so much more in our peripheral vision, at a much higher bandwidth.
Some would argue that's an indication that AR will happen, but it's still so low deff, and incredibly intrusive, as much as I love the demos and the vision (pun not intended) behind it.
As far as I can see, the only motivation for the visual overall is that they need something to fill the gap until they have some real AI innovations to show. This is a "tick" in the traditional "tick" -> "tock" development and release cycle - a facelift while they work on some difficult re-engineering underneath. But that's not AR, it AI.
You will need to use an unrolled main loop similar to what we have in PGlite, using the "single user mode" (you likely don't want to fork sub processes like a normal Postgres). The problems come with then trying to run multiple instances in a single process, Postgres makes heavy use of global vars for state (they can as they fork for each session), these would clash if you had multiple instances. There is work happening to possibly make Postgres multi-threaded, this will solve that problem.
The long term ambition of the PGlite project is to create a libpglite, a low level embedded Postgres with a C api, that will enable all this. We quite far off though - happy to have people join the project to help make it happen!
I work on PGlite, we have an experimental WASI build that can run in a WASI runtime, it should enable dropping the Node requirement. It lacks error handling at the moment (WASM has no long jump, and Postgres uses that for error handling - Emscripten has hacks that fix this via JS), and so we haven't yet pushed it far.
Do ping me on our Discord and I can point you towards it.
Happy to answers any PGlite questions while I'm here!
- know the total length of the tape, and when it's changed, and therefore predict the current radius of the roll
- a small sprung variable resistor pressing against the under side of the role
But that could be over-engineering something that almost more perfect if it's a little imperfect.
Blindly opening "PRs" (Postgres doesn't work in this way, it's diffs on a mailing list) would not get you anywhere very fast. You need buy in and consensus from the wider development team.