Hacker News Reader

Comment by samwillis | Hacker News Reader

Hey everyone, I work on PGlite. Excited to see this on HN again.

If you have any questions I'll be sure to answer them.

We recently crossed a massive usage milestone with over 3M weekly downloads (we're nearly at 4M!) - see https://www.npmjs.com/package/@electric-sql/pglite

While we originally built this for embedding into web apps, we have seen enormous growth in devtools and developer environments - both Google Firebase and Prisma have embedded PGlite into their CLIs to emulate their server products.

View on HN

This looks really interesting...but why WASM-only? Naively it seems like WASM-ification would be a 2nd step, after lib-ification.

Obviously missing something...

WASM means you only need to develop for one target run time. That's my guess as to why.

If I understand correctly, what this project does is take the actual postgresql sources, which are written in C, compile them to wasm and provide typescript wrappers. So you need the wasm to be able to use the C code from js/ts.

Yes. I would like to use the code as a library from something other than js/ts.

So compile it and use it?

Sounds you only need to create the APIs for calling into WASM if so, so as long as your language of choice can do that, you're good to go.

That adds extra unnecessary complexity. The code is written in C. There are C compilers for all CPUs. So just call the C code from <other language that's not JS>.

Well, a project has scope.

Looking at the repo, it started as postgres-in-the-browser. An abstract interface with C and wasm as targets is just more scope.

But it looks like the hard part of patching postgres to librar-ify it is already done agnostically in C.

So you just need to ctrl-f for "#if defined(__EMSCRIPTEN__)" to impl those else branches and port the emmake file to make.

You can use it in Rust if you like. I've used pglite through wasmer before. Also [pglite-oxide](https://lib.rs/crates/pglite-oxide) is pretty usable.

Yeah... I was super excited by this project when it was first announced--and would even use it from Wasm--but since it ONLY works in Wasm, that seemed way too niche.

Hi there, would you like to share the progress of converting PGlite into a native system library? I can see there is a repo for that, but it hasn't been updated for 5 months

We are actively looking into it. But as you can see from the comments here, there are quite a lot of other features that users want and we have limited bandwidth. We will do it!

This is awesome, thanks for your work! Could this work with the file system api in the bowser to write to user disk instead of indexeddb? I'm interested in easy ways for syncing fot local-first single user stuff <3 thanks again

That's a very nice idea, we will look into it!

I see you guys are working on supporting the postgis extension. This would be HUGE!!! The gis community would be all over this.

If anyone wants to help out who has compiled the postgis extension and is familiar with WASM. You can help out here. https://github.com/electric-sql/pglite/pull/807

Thanks for your work!

Is the project interested in supporting http-vfs readonly usecases? I'm thinking of tools like DuckDB or sql.js-httpvfs that support reading blocks from a remote url via range requests.

Curious because we build stuff like this https://news.ycombinator.com/item?id=45774571 at my lab, and the current ecosystem for http-vfs is very slim — a lot of proofs of concept, not many widely used and optimized libraries.

I have no idea if this makes sense for postgres — are the disk access patterns better or worse for http-vfs in postgres than they are in sqlite?

This looks REALLY awesome. Could you name a few usecases when i would want to use this. Is the goal to be an sqlite/duckdb alternative?

Any chance for a Flutter library?

I'm interested to use Pglite for local unit-testing, but I'm using timescaledb in prod, do you think you will have this extension pre-built for Pglite?

I'm not aware of anything trying to compile timescale for it. Some extensions are easer than other, if there is limited (or ideally no) network IO and its written in C (Timescale is!) with minimal dependencies then its a little easer to get them working.

I’ve had incredible success with testcontainers for local unit-testing

We have a walk-through on porting extensions to PGlite: https://pglite.dev/extensions/development#building-postgres-...

Does pglite in memory outperform “normal” postgres?

If so then supporting the network protocol so it could be run in CI for non-JS languages could be really cool

Look into libeatmydata LD_PRELOAD. it disables fsync and other durability syscalls, fabulous for ci. Materialize.com uses it for their ci that’s where i learned about it.

for CI you can already use postgresql with "eat-my-data" library ? I don't know if there's more official image , but in my company we're using https://github.com/allan-simon/postgres-eatmydata

You can just set fsync=off if you don't want to flush to disk and are ok with corruption in case of a OS/hw level crash.

Huh, i always just mounted the data directory as tmpfs/ramdisk. Worked nicely too

Yupp, this has big potential for local-first !

Small world! We spoke about this at the QCon dinner.

Amazing work! It makes setting up CI so much easier.

huh. could you tell how you use it in ci?

I'm using it for a service that has DB dependencies. Instead of using SQLite in tests and PG in production, or spinning up a Postgres container, you use Postgres via pglite.

In my case, the focus is on DX ie faster tests. I load shared database from `pglite-schema.tgz` (~1040ms) instead of running migrations from a fresh DB and then use transaction rollback isolation (~10ms per test).

This is a lot faster and more convenient than spinning up a container. Test runs are 5x faster.

I'm hoping to get this working on a python service soon as well (with py-pglite).

Thank you for the details. This makes a lot of sense!

Well downloads doesn’t equal usage does it ?

How do you know how many deployments you actually have in the wild?

True downloads don’t equal usage but there’s a correlation. I also doubt deployment equals usage - I can deploy to some env and not make any requests.

Additionally, how you can get data on how many deployments without telemetry? The only telemetry that I’m interested in is for my uses, and don’t really care about sending data on deployment count to a third party. So the download count becomes a “good enough” metric.

Reply on news.ycombinator.com