Hacker News Reader

Comment by tptacek | Hacker News Reader

Just a note that LiteFS isn't a distributed filesystem; despite the name, it's not a filesystem at all, but rather just a filesystem proxy. It does essentially the same thing that the VFS layer in SQLite itself does, but it does it with a FUSE filesystem, so you don't have to change SQLite's configuration in your application. As for the "distributed" part of it: LiteFS has a single-writer multi-reader architecture; it's the same strategy you'd use to scale out Postgres.

It's a little ironic to see LiteFS brought up in relation to a SQLite fork, since the premise of LiteFS is not changing the SQLite code you link into your application. Much of the work in LiteFS is about cooperating with standard SQLite.

At any rate: it seems somewhat unlikely that a hard fork of SQLite is going to succeed, in that part of the reason so many teams are building SQLite tooling is the trust they have in the current SQLite team.

View on HN

>LiteFS has a single-writer multi-reader architecture; it's the same strategy you'd use to scale out Postgres

I'm curious about how such a strategy deals with applications that update a value and then read the updated value back to the user, since there might be a replication delay between the write (that goes to the master) and the read (that comes from the closest replica). Do you make optimistic updates on the client or do you club (write-then-read) operations into a transaction?

LiteFS author here. It depends on the application. You can check the replication position on the replicas and simply wait until the replica catches up. That requires maintaining that position on the client though.

An easier approach that works for a lot of apps is to simply have the client read from the primary for a period of time (e.g. 5 seconds) after they issue a write. That's easy to implement with a cookie and it's typically "good enough" consistency for many read-heavy applications.

This is an application (and frequently data type) dependent decision. Some data is safe to return on acceptance others need to wait for acknowledged writes.

The most naive solution is to just make all writes slow but acknowledged.

Has sqlite been forked before? This is the first true fork and re-license attempt that's caught my eye. The others I've seen are "ports" and "modifications", but always pointing people back upstream.

It's possible that the appetite for a SQLite fork is there, but nobody has provided it.

> Has sqlite been forked before?

There is a fork called SQLCipher with native encryption. Probably lots of others too, but this one is the one that I remember.

I do remember the author of LMDB [0] porting some parts of sqlite to use lmdb instead, and then talking about the results. A quick googling doesn't seem to give me a result though.

[0] - http://www.lmdb.tech/doc/

https://github.com/LMDB/sqlightning

https://github.com/LumoSQL/LumoSQL

TFA lists forks. rqlite comes to mind.

rqlite[1] author here. I wouldn't consider rqlite a fork in any sense, just so we're clear. That rqlite uses plain vanilla SQLite is one of its key advantages IMHO. Users have no concerns they're not running real SQLite source.

That said, there are some things that would be much easier to do with some changes to the SQLite source. But I think the message that rqlite sits on top of pure SQLite makes is still the right choice.

[1]: https://github.com/rqlite/rqlite

My mistake. Thanks for the correction!

Is rqlite a fork? It strikes me more as an application that uses SQLite as a dependency, rather than a fork of SQLite itself.

Nope, as explained in my other comment. It is, as you say, built on top of SQLite.

> LiteFS has a single-writer multi-reader architecture;

Note that SQLite transactions are also fundamentally single writer, multiple reader (isolation level serializized with snapshot isolation and no work-forking).

So if you want to make SQLite distributed you will have to end up with not just a single writable replicate but a global write log on transaction level. Which is very very bad for performance for many use cases. (E.g. in PostgreSQL if you use serialized & snapshot transaction isolation it's "forking the world" for parallel write transactions and if multiple transactions have no overlap they can complete in parallel without a problem (in parallel but on the same single write enabled replica). This is good enough for quite a bunch of applications and can still have a decent throughput).

As far as I can tell many use-cases which could profit from a distributed SQLite would not really like a per-transaction global lock but would be okay with something like what Postgres does. Through there are always some exceptions.

Reply on news.ycombinator.com