30,161 karma · joined March 17, 2008
Working on sync engines, PGlite, and Tanstack DB at Neon/Databricks. Previously co-founder ElectricSQL (acquired by Databricks).
sam dot willis at gmail dot com
samwillis.co.uk
Twitter: @samwillis
Of course they considered it, but the tradeoffs didn't match what they wanted to do - plus they found you could scale to this level without sharding.
What you have to remember is that for many teams, when their product takes off, they are not equipped with the deep internal knowledge of how to scale a particular part of their stack. This was an awesome story from a small team having to tackle those challenges, and how they were learning as they went. So, while there are some of those "can't you just", and "whats interesting about this?" comments here, with the narrative of the growth rate, and the very high profile of the product, it was the perfect user talk for a internal development focused conference.
The key insight, and main message, of the talk was that if you are not too write heavy you can scale Postgres to very high read throughput with read replicas and only a single master! That is exactly the message that needs to be spelled out as that covers the vast majority of apps.
As an observation, in the Q&A at the end of the talk the questions, primarily from core Postgres developers, were focused on learning about the use case, and not an opportunity to suggest that they were doing anything wrong (not quite the same as this thread could get). A genuinely awesome group of very friendly and welcoming people in the Postgres community.
> Local-first software enables greater longevity because your data, and the software that is needed to read and modify your data, are all stored locally on your computer.
There are many ways to achieve this goal - open document standards, open source servers, a escrowed release of the server, or this idea of a bail out system for taking the current version and self hosting it. All are commendable.
Actually achieving all seven ideals is hard, and there isn't much modern software that does it. But in my view anyone trying to achieve even a few of the ideals is making strides towards a worthy goal.
There is a lot of exciting stuff happening in the local-first software movement at the moment, and a lot of that is related the sync engines that are being built (disclaimer: I work for ElectricSQL, we are building one). These sync engine don't inherently make your software local-first, fitting all the ideals, but they do make it a lot easer to do so. They are an important building block. But there is more needed, we need more open document standards - Automerge, Yjs, Loro and the other CRDT data structures are perfect lower level data structures to build these higher level abstractions on. Martin Kleppmann has talked quite a bit about sync engines that are disconnected from the underlying application, essentially pluggable sync engine, you choose who to use to sync you copy of a document or application - I'm excited to see where this goes.
But, we also need to free up the application distribution platform. The app stores are walled gardens that prevent some business models, native apps (while performant) are tied to a specific platform (or even version!), and the web is inherently tied to servers and browsers. The web platform though, i.e. JS, HTML, CSS, is perfect for building high longevity software that runs anywhere, on any device, the issue is the distribution. We need a middle ground, an application package that built on web standards, but isn't tied to a server. I want to download a bundled app to my machine and have a copy, email to a family member, or even open and hack on it. Thats the final missing piece.
A downloadedable app, with an open and pluggable sync engine would achieve the same goles of this ejectable idea.
Describe what you want, get it to confirm a plan, ask it to start, and go make coffee.
Come back 5min later to 1k lines of code plus tests that are passing and ready for review (in a nice code review / diff inline interface).
(I've not used copilot since ~October, no idea if it now does this, but suspect not)
I'm also aware of at least a handful of hacker new members, and people I've followed, who took time out of their carriers to do a "tour of duty" at 18f.
I feel for all today, you were doing good work!
Fireworks could have been Figma, it could have been the default platform every designer used. But Adobe didn't understand it, they saw it as a weird Photoshop competitor and shelved it.
The last few versions bundled with CS were clearly neglected maintenance releases. I finally stopped using my slowing rotting copy in about 2014 when I got a Mac with a retina screen and fireworks was stuck with a terrible pixel doubled ui. :-(
Throw a Caddy reverse proxy in front of your normal dev server and you immediately get HTTP2 via the root certificate it installs in your OS trust store. (https://caddyserver.com/docs/automatic-https)
We (ElectricSQL) recommend it for our users as our APIs do long polling, which with HTTP2 doesn't lock up those 6 concurrent connections.
I've also found that placing it in front of Vite for normal development makes reloads much faster. Vite uses the JS module system for loading individual files in the browser with support for HMR (hot module replacement), this can result in a lot of concurrent requests for larger apps, creating a queue for those files on the six connections. Other bundlers/build tools bundle the code during development, reducing the number of files loaded into the browser, this created a bit of a debate last year on which is the better approach. With HTTP2 via Caddy in front of Vite you solve all those problems!
Really interesting data, particularly when you compare the very low level of outages in Canada/UK compared with the US.
I violated my own rule of always re-reading a post 5 min after posting it...
But is suppose many organisations just don't give people the autonomy and authority to do such tings.
It's possible to use PGlite from any language using native Postgres clients and pg-gateway, but you lose some of the nice test DX when it's embed directly in the test code.
I'm hopeful that we can bring PGlite to other platforms, it's being actively worked on.
The other thing I hope we can look at at some point is instant forks of in memory databases, it would make it possible to setup a test db once and then reset it to a known state for each test.
(I work on PGlite)
I've joked before that the last generation of human machine interfaces ware invented at Xerox park, and the next generation is being invented at TLDraw of Finsbury Park. But it's not really a joke, I genuinely believe it.
There was a previous attempt to bring async io to Postgres, but sadly it went dormant: https://commitfest.postgresql.org/34/3316/
A more recent proposal was to make it possible to swap out the storage manager for a custom one without having to fork the codebase. I.e. extensions can provide an alternative. https://commitfest.postgresql.org/49/4428/
This would allow for custom ones that do async IO to any custom storage layer.
There are a lot of interested parties in the new proposal (it's come out of Neon, as they run a fork with a custom storage manager). With the move to separate compute from storage this becomes something many Postgres orgs will want to be able to do.
A change of core to use async io becomes slightly less relevant when you can swap out the whole storage manager.
(Note that the storage manager only handles pages in the heap tables, not the WAL. There is more exploration needed there to make the WAL extendable/replaceable)
Live demo: http://linearlite.examples.electric-sql.com
Code: https://github.com/electric-sql/electric/tree/main/examples/...
It also allows us to implement a VFS layer underneath PGlite in a native mode. So things like the in-memory VFS, or a custom VFS, would be possible.
We have not committed to one route or the other yet.
- the WASI build is targeting WASI snapshot preview 1, and so should work in any compatible runtime.
- for the web, currently we use Emscripten, but we are considering moving to a single WASI build there as well. We'll be writing our own JS WASI shim if we do. Having full control of the JS code will help to solve some of the problems we've faced with Emscripten.
- we are also exploring a route to native where we take the WASI build and decompile it back to C. This seems a little mad, but it makes it possible to compile (with any tool chain) a native binary with a very minimal WASI-like api that can be linked to from any app. It essentially end up a little like the SQLite amalgamated header file as a build route. It's very experimental, and we haven't committed to it yet, but it looks like it may work.
Musk Inc. is, in essence, a privet equity company. He raises a fund, and invests it along with his own cash into a business he knows well, and with which he believes he can disrupt the status quo.
He's very hands on, understanding all the important core details, setting culture, and pushing hard. But he also delegates to experts he brings in to run his businesses.
It's obviously not the same as PE, but there are distinct similarities. With each of his companies he clearly can't take an active role all the time, but what his team are experts at is identifying the areas he needs to be on top of, and they will quite literally fly him in for it. It seems to me it's always the start of something, be it a new company or project. He will be there 24/7 getting it off the ground, but then hand over to lieutenants to run.
It's a formula that seems to work again and again. We're (well those of you in the US) are in for an interesting time over the coming year as he has a new project to kick start.
Worth reading the thread, there are some good insights. It looks like he will be waiting on Postgres to take the initiative on implementing this before it makes it into a release.
It's interesting to consider the question of what this would look like if you put Postgres, rather than SQLite, in process? With PGlite we can actually look at it.
I'm hesitant to post a link to benchmarks after the last 24 hours on Twitter... but I have some basic micro-benchmarks comparing WASM SQLite to PGlite (WASM Postgres that runs in-process): https://pglite.dev/benchmarks#round-trip-time-benchmarks
It's very much in the same ballpark, Postgres is a heavier databases and is understandably slower, but not by much. There is a lot of nuance to these benchmarks though as the underlying VFSs are a little different to each other, and PGlite has a WAL rather then SQLite which is in its rollback journal mode (I believe this is why PGlite is faster for some inserts/updates).
But essentially I think Postgres when in-process would be able to perform similarly to SQLite with many small queries and embracing n+1. But having said that I think the other comments here about query planning are important to consider, if you can minimise your queries, you minimise the scans of indexes and tables, which is surely better.
What this does show, particularly when you look at the "native" comparison at the end, is that removing the network (or at least a local socket) from the stack being the two closer together.
The pg_duck project has the eventual aim to implement a column storage engine for Postgres. There are a few steps to get there as it needs to be tied into the Postgres page storage and replication system. So it's not solved by the first version of pg_duck, but the team is incredible and I believe it will happen.
Neon and Oriole (acquired by Supabase) are both open source and separate storage and compute. There is a few steps more for them to go to be truly usable self hosted, but they will get there, and some of the work they are doing will hopefully be upstreamed.
It's crazy what you can achieve with typescript types, what about inferring sql query results types from a raw sql query string: https://github.com/nikeee/sequelts
We have a very different model for access to data produced by government agency use to that in the US.
USGS Topographic maps: public domain / free
UK OS Topographic maps: paid access, and it's not cheep
US National Weather Service: Public domain / free commercial use
UK MetOffie: Payed access for commercial use
A clean rewrite of a wireless networking stack that does real-time synced audio streaming and device discovery on any users home WiFi (which is notoriously bad!) is a decision to face a lot of pain finding the edge cases that were previously fixed 10y ago.
Essentially they replaced the 15 year old, and incredibly robust network based on UPnP, with a new system using mDNS+HTTP+Websockets. And at the same time stuck their cloud in the loop for connecting to third party music services.
It's resulted in a much more fragile networking stack, far more prone to incompatibilities with users WiFi configurations. Which due to being so much heavier, some of their older hardware may struggle with. And that's all before you consider the UX changes in the app, and the missing functionality (how do you release a forced upgrade before finishing all old, and essential, functionality?!)
I assume that they have started removing some/all of the old code from the speakers which makes the old app incompatible.
What I would love is a Postgres task queue that does multi-step pipelines, with fan out and accumulation. In my view a structured relational database is a particularly good backend for that as it inherently can model the structure. Is that something you have considered exploring?
The one thing with listen/notify that I find lacking is the max payload size of 8k, it somewhat limits its capability without having to start saving stuff to tables. What I would really like is a streaming table, with a schema and all the rich type support... maybe one day.
(I work on PGlite)
People often mistakingly describe the vdom as faster than the dom, this is incorrect. It would be faster than throwing away the whole components dom and rebuilding, so the same templating code building a new dom, rather than a vdom that's then diffed. Hand crafter mutations will be faster than a vdom diff, simply because the computer is doing les work, however much more error prone.