Show HN: TonboLite – Scale SQLite with S3, Minimize ETL

Hi! I am Tzu and the team from Tonbo here.

TonboLite: https://github.com/tonbo-io/tonbolite is a SQLite extension based on Tonbo: https://github.com/tonbo-io/tonbo. It enables SQLite to create tables suitable for analytical processing on target platforms like WebAssembly in browser and efficiently write data. The data in the tables is organized as tiered Apache Parquet format files, stored on demand either on local disks (using OPFS as native I/O) or object storage services (such as S3). You can use it by creating virtual tables in regular SQLite.

TonboLite started with the exploration of Tonbo application. The goal of Tonbo is to write data for analytical processing (like log processing, metrics monitoring, or text search) to unlimited remote storage in SQLite or PostgreSQL.

We tried SQLite as it is the most popular transactional database for the edge. One of the most requested improvements for SQLite is better support for append-only writes (e.g., logs, time-series data), which are common in analytical data. Append-only writes present two main challenges for SQLite: 1. SQLite3 uses B-Trees as its storage structure. B-Trees perform well for random access but are not optimized for write efficiency in append-only scenarios like log-structured databases. 2. Append-only data often involves large write volumes, making it difficult to store extensive historical data in a single-machine environment long-term. However, Tonbo is well-suited to address these two issues: 1. Tonbo organizes Parquet files into an LSM Tree, a database structure highly optimized for concurrent append-only writes. 2. Tonbo supports gradually writing historical files in the background to horizontally scalable storage services (such as S3).

This project is still in the very early stages. If you are interested in the project's goals and future developments, welcome to join the discussion: https://discord.gg/j27XVFVmJM. We are happy to answer any questions you may have.