avinassh

> One of the nice things about sqlite is that there is a very extensive test suite that extensively tests it.

Yes, that sets a high bar for us. We plan to use Deterministic Simulation Testing and Antithesis to reach the rigorous testing standards of SQLite.

Limbo comes with a simulator too

> My solution was to have a central function on a timer that read and parsed the files every couple of minutes, loaded the data into a SQLite database, indexed it, and put the file in S3.

I really love using S3 as a storage backend!

> Let's async all the things IO and not pay attention to database integrity and reliably fsync'ing with storage

I am not sure how does this affect database integrity or reliably fsync-ing

for e.g. TigerBeetle is another rock solid database which uses async IO. I mentioning it because it is way more mature than Limbo and does a great job at durability

> The web site says "the project does not accept patches from people who have not submitted an affidavit dedicating their contribution into the public domain."

I have been always curious about this. Is there any more public information to this? When one submits a affidavit, do all their work become public domain? Do you highlight the code and get a affidavit with each contribution?

for e.g. in my country India, I don't think it is not possible to get such Govt approved affidavit.

disclosure: I work at Turso

> code of ethics as almost toxic

This is simply not true. Can you tell me where it is being said so?

> then folding “libSQL into the Turso family” within the year.

libSQL was always part of Turso. So, I don't get your point.

> They forked, played politics, added a few features (with some ill-considered incompatibilities), and properly documented zero of them.

Again this is incorrect. There are some docs here: https://github.com/tursodatabase/libsql/tree/main/docs

I am really not sure why are you so angry about libSQL.

I am the author of this blog post and I didn't expect to see it on the front page! For disclosure, I work at Turso and one of the authors, Pekka, is from Turso.

This paper came out in April 2024 when Limbo was in its nascent stages. It has seen many improvements since then, one being support for Deterministic Simulation Testing.

repo: https://github.com/tursodatabase/limbo

hey, thats fair. I have mentioned that I work at Turso in my blog's about page, but I don't expect everyone to check that. I have updated the post to include a disclosure, thanks!

As other commenters have already pointed out, SQLite does not take outside contributions.

We already have a fork, called libSQL. However, the goals of Limbo are far more ambitious and we cannot rewrite some parts step by step. We want to have DST ( Deterministic Simulation Testing), a testing methodology pioneered by Foundation DB and TigerBeetle. It is not easy to do that in an existing codebase

> In any case, the programming language should have little to no impact on the database performance, since the majority of the time is spent waiting on io anyway

may be! However, Rust makes some things easier. It is also easy to maintain and reiterate

As of now it has a single writer, same like SQLite. But we plan to add MVCC with multiple writers in the future. Pekka has experimented with MVCC earlier: https://github.com/penberg/tihku

disclosure: I work here. I am happy to answer any questions

tl;dr We are rewriting SQLite in Rust. It uses Asynchronous I/O, considers WASM as first class, and has Deterministic Simulation Testing support from the beginning.

source: https://github.com/tursodatabase/limbo

it does now: https://github.com/dbeaver/dbeaver-jdbc-libsql

(author here)

definitely!

I plan to add a batch write API. Also, an API where it buffers till it reaches certain size or a timeout to write to S3

tracking the batch write here: https://github.com/avinassh/s3-log/issues/3

(author here)

Currently, there is durability guarantee. The call returns only after successful write to S3. This is close to fsync in a single node system. I plan to add a relaxed write mode

I am tracking the issue here: https://github.com/avinassh/s3-log/issues/4

crazy idea, but SQLite

Hey! I am @avi.im

I mostly post stuff about databases, distributed systems and my frustrations with Rust

https://bsky.app/profile/avi.im

perfect. beautifully designed. I will try this for my next side project.

> .::/

whats this magic

You would be surprised :)

A couple of my friends who work in the financial domain showed surprise at this post - they had no idea this was possible. I guess most people just assume databases 'just work'?

SQLite or any other mainstream database like Postgres don't do checksums. Wouldn't you be concerned if you are using it for financial data or medicine or something crucial?

> I feel it's quite unfair for the title to call out sqlite on checksums, when in reality, as the very closing line of the article states most databases don't do checksum verifications.

Hi, author of the post here. I work mostly with SQLite and that's the database I'm most familiar with, hence mentioning it in the title. I have also noted at the bottom:

> Again, this is not a bug. Most databases (except a few) assume that the OS, filesystem, and disk are sound. Whether this matters depends on your application and the guarantees you need.

now I wrote a follow up post - https://avi.im/blag/2024/databases-checksum

is there any page which explains how does Accelerate works? I am interested knowing the internal technical details

I love one Raindrop’s feature where it lets you bookmark all the tabs in a window

Hey! I am @avi.im

I mostly post stuff about databases, distributed systems and my frustrations with Rust

https://bsky.app/profile/avi.im

checkout Simon Willison's notes if you want to know how to get this running and try: https://til.simonwillison.net/sqlite/compile-sqlite3-rsync

related: any high level architecture details on how BlueSky uses SQLite per user?

I'd love to know how they have hooked VFS with WAL to monitor changes. The SQLite's WAL layer deals with page numbers where as VFS deals with file and byte offsets. I am curious to understand how they mapped it, how they get new writes to the WAL and read from the WAL.

it does mention it:

there’s just 1 static binary, all I need to do to deploy it is copy the binary. If there are static files I can just embed them in the binary with embed.

Is the benchmark code available somewhere / open source?

last I checked, Substack did not allow writers to set canonical links for their posts.