HNHacker News
TopNewBestAskShowJobs

russellthehippo

310 karma · joined January 11, 2017

submissionscomments
russellthehippo··on Show HN: Turbolite – a SQLite VFS serving sub-250ms cold JOIN queries from S3
No. Turbolite is explicitly read-write, and it’s genuinely hard to balance availability against local query speed. Those tradeoffs are real. I don’t think your response is describing this project very accurately.
russellthehippo··on Show HN: Turbolite – a SQLite VFS serving sub-250ms cold JOIN queries from S3
rqlite does this well but it’s still server based
russellthehippo··on Show HN: Turbolite – a SQLite VFS serving sub-250ms cold JOIN queries from S3
Thanks! I think the hole punching is the key one, as it is important for "this delete has happened but we need to free the space without a vacuum".
russellthehippo··on Show HN: Turbolite – a SQLite VFS serving sub-250ms cold JOIN queries from S3
Yeah I mostly agree, and another comment brought up this idea kind of. If you can keep SQLite local, that’s usually the better answer. Local reads are insanely hard to beat, and Litestream is the canonical “keep SQLite local and ship durability elsewhere” model.

That’s actually a big part of why I started another project, haqlite: https://github.com/russellromney/haqlite which is much more on the “keep SQLite local” side: leader election via S3 conditional PUTs, WAL replication to S3, follower catchup, write forwarding to the leader, and graceful leader handoff. So it can get pretty close to zero-downtime deploys with 2+ nodes.

But not true zero downtime on a single server — if there isn’t already another warm follower alive, there is nowhere for leadership to go. So in my head:

- Litestream: local SQLite is primary; object storage is for durability / replication / failover

- haqlite: kind of a Litestream but with a focus on embedding it into any application.

- turbolite: object storage is the actual backing store, and the question is how plausible cold reads can get. No HA ideas

russellthehippo··on Show HN: Turbolite – a SQLite VFS serving sub-250ms cold JOIN queries from S3
Couldn't have said it better myself.
russellthehippo··on Show HN: Turbolite – a SQLite VFS serving sub-250ms cold JOIN queries from S3
Cost really depends on your workflow, your durability needs, prefetch aggression (how quickly you need all the data locally), and how aggressively you cache. The storage is what it is, and it's cheap. The variation comes in transaction counts.

If you evict the cache on every read/write, then all reads and writes do S3 GET. PUTs are 10-100x more expensive than GETs usually. Check the benchmark/README.md for GET counts, but it's usually 5-50 per cold read (with interior B-tree pages on disk).

S3 GETs are $0.0004/1000, S3 Express is $0.00003. So 10 queries per minute all day long averaging 20 GET operations, with full eviction on each request, would be 20*10*$0.0004*60*24/1000*30 = $3.45 per month. With S3 Express One Zone that's $0.26/mo. Both plus storage costs, which would probably be lower.

On Wasabi, that would be just the cost of storage (but they have minimum 1TB requirements at $6.99/mo).

If you checkpoint after every write and write 10 times per minute, each write hitting e.g. 5 page groups (S3 objects) plus the manifest file, the analysis looks like: 6*10*$0.005*60*24/1000*30=$12.96. Again that's worst case for the benchmark 1.5GB database. On S3 Express that' $2.92.

Point is - it's not too bad, and that's kind of worst-case scenario where you evict on every request every 6 seconds which isn't really realistic. If you evict the cache hourly, that cost goes is 1/600th - less than a $0.01 per month.

Summary: use S3 Express One Zone, don't evict the cache too often, checkpoint to S3 once a minute (turbolite lets you checkpoint either locally (disk-durability) or locally+S3 separately), and you're running a decent workload for pennies every month.

[Apologies for the spreadsheet math in plaintext]

russellthehippo··on Show HN: Turbolite – a SQLite VFS serving sub-250ms cold JOIN queries from S3
It's funny, I did consider that and put some work into doing it. I moved that work into a different project, https://github.com/russellromney/haqlite that focuses on high-availability SQLite. Like Litestream but focused on embedding in your application. It uses conditional PUTs for leader election, ships WAL to S3, and writes (and reads for consistency) are forwarded to the leader - so it's still single writer, but failover happens automatically when the leader fails to claim the leader lease again. Data window is the lease timeout + checkpoint interval. Maybe I'll explore how to use haqlite + turbolite together.

Short answer: conditional PUTs for distributed scare me for multiwriter. The issue isn't doing the writes, it's ensuring that the writer is writing against the most current data. For OLTP workloads with upserts, this is very hard! If you have immutable data without any upserts and your writes don't depend on reads, that actually works really well. But in any other scenario it's dangerous. The writer needs to ensure that the other writers have checkpointed first, and there's not a great way to do that.

One thing that could make this work in turbolite is a fast distributed lock system with transaction and lock timeouts. E.g. use Redis as the lock lease holder, and distributed writers acquire it, fetch the latest manifest from s3, sync any data they need, do the write, checkpoint, update the lock to be released and note the new manifest version, and return success to the user. then before any reads, they simple check Redis for the new manifest (or S3 for guarantee that the manifest update/lock release didn't fail). All writes have an N second timeout, and the write lock has N second + T timeout, so it's guaranteed that successful checkpoints are used in the next read, as long as readers check the manifest first.

This could work, but it's <still> single writer lol. But you'd only want it with infrequent writes. And reads cost an S3 GET. So I guess it would work best with Wasabi, which doesn't charge for operations, or self-hosted MinIO.

russellthehippo··on Show HN: Turbolite – a SQLite VFS serving sub-250ms cold JOIN queries from S3
Also it’s super easy to add them, it’s definitely on the roadmap. The manifest tracks the data locally and could quickly evict on any operation that accesses it. Very safe post-checkpoint
russellthehippo··on Show HN: Turbolite – a SQLite VFS serving sub-250ms cold JOIN queries from S3
Thanks for your kind words!
russellthehippo··on Show HN: Turbolite – a SQLite VFS serving sub-250ms cold JOIN queries from S3
Very cool, thanks. I hadn’t seen Graft before, but that sounds pretty adjacent in a lot of interesting ways. I looked at the repo and see what I can apply.

I've tried out all sorts of optimizations - for free pages, I've considered leaving empty space in each S3 object and serving those as free pages to get efficient writes without shuffling pages too much. My current bias has been to over-store a little if it keeps the read path simpler, since the main goal so far has been making cold reads plausible rather than maximizing space efficiency. Especially because free pages compress well.

I have two related roadmap item: hole-punching and LSM-like writing. For local on non-HDD storage, we can evict empty pages automatically by releasing empty page space back to the OS. For writes, LSM is best because it groups related things together, which is what we need. but that would mean doing a lot of rewriting on checkpoint. So both of these feel a little premature to optimize for vs other things.

russellthehippo··on Show HN: Turbolite – a SQLite VFS serving sub-250ms cold JOIN queries from S3
Great question. I have some eviction functions in the Rust library; I don’t expose them through the extension/VFS yet. The open question is less “can I evict?” and more “when should eviction fire?” via user action, via policy, or both.

The obvious policy-driven versions are things like:

- when cache size crosses a limit

- on checkpoint

- every N writes (kind of like autocheckpoint)

- after some idle / age threshold

My instinct is that for the workload I care about, the best answer is probably hybrid. The VFS should have a tier-aware policy internally that users can configure with separate policies for interior/index/data pages. But the user/application may still be in the best position to say “this tenant/session DB is cold now, evict aggressively.”

russellthehippo··on Show HN: Turbolite – a SQLite VFS serving sub-250ms cold JOIN queries from S3
Also I want to acknowledge the other projects in adjacent parts of this space — raw SQLite range-request VFSes, Litestream/LiteFS-style replication approaches, libSQL/Turso, Neon, mvsqlite, etc. I took a lot of inspiration from them, thanks!
russellthehippo··on Show HN: Turbolite – a SQLite VFS serving sub-250ms cold JOIN queries from S3
A bit more color on what I found interesting building this:

The motivating question for me was less “can SQLite read over the network?” and more “what assumptions break once the storage layer is object storage instead of a filesystem?”

The biggest conceptual shift was around *layout*.

What felt most wrong in naive designs was that SQLite page numbers are not laid out in a way that matches how you want to fetch data remotely. If an index is scattered across many unrelated page ranges, then “prefetch nearby pages” is kind of a fake optimization. Nearby in the file is not the same thing as relevant to the query.

That pushed me toward B-tree-aware grouping. Once the storage layer starts understanding which table or index a page belongs to, a lot of other things get cleaner: more targeted prefetch, better scan behavior, less random fetching, and much saner request economics.

Another thing that became much more important than I expected is that *different page types matter a lot*. Interior B-tree pages are tiny in footprint but disproportionately important, because basically every query traverses them. That changed how I thought about the system: much less as “a database file” and much more as “different classes of pages with very different value on the critical path.”

The query-plan-aware “frontrun” part came from the same instinct. Reactive prefetch is fine, but SQLite often already knows a lot about what it is about to touch. If the storage layer can see enough of that early, it can start warming the right structures before the first miss fully cascades. That’s still pretty experimental, but it was one of the more fun parts of the project.

A few things I learned building this:

1. *Cold point reads and small joins seem more plausible than I expected.* Not local-disk fast, obviously, but plausible for the “many mostly-cold DBs” niche.

2. *The real enemy is request count more than raw bytes.* Once I leaned harder into grouping and prefetch by tree, the design got much more coherent.

3. *Scans are still where reality bites.* They got much less bad, but they are still the place where remote object storage most clearly reminds you that it is not a local SSD.

4. *The storage backend is super important.* Different storage backends (S3, S3 Express, Tigris) have verg different round trip latencies and it's the single most important thing in determining how to tune prefetching.

Anyway, happy to talk about the architecture, the benchmark setup, what broke, or why I chose this shape instead of raw-file range GETs / replication-first approaches / etc.

russellthehippo··on Show HN: LLM Skirmish – a benchmark where LLMs play RTS games, by writing code
Whoa, this is sick. Like adversarial chess training but inverted for model evaluation. The model has to be both correct and fast at code while managing tactics and strategy well. I wonder if it should extend to general-soldier models, like an agent swarm. obv would kill tokens but would be super interesting
russellthehippo··on Late night pizzeria nearby The Pentagon has suddenly surged in traffic
Confirmed - strike on Venezuela and capture of Maduro. https://www.cnbc.com/2026/01/03/trump-us-operation-captured-...
russellthehippo··on Elevated errors across many models
Anthropic is very focused on AI safety. It makes LLMs safe by shutting down anyone from using them
russellthehippo··on Kuzu Graph DB is archived:(
Super sad to see this project get archived. I've really enjoyed working with it, it's been an amazing addition to projects where node-based data structures are obviously the move but I don't want to use server-based approaches.
russellthehippo··on What Are MCP Servers?
https://archive.ph/5b4wu
russellthehippo··on What Are MCP Servers?
Answer - MCP servers are APIs, basically. Nice.
russellthehippo··on Bauplan – Git-for-data pipelines on object storage
Congrats on the more official launch! Super promising, first product that shares dbt-type data organization/orchestration capabilities with a compute layer worthy of replacing existing data warehouses/python environments.
russellthehippo··on [dead]
nice
russellthehippo··on Reliability: It’s not great
Agree - V2 apps on machines are incredibly slick to launch (create/start/stop), get info on with graphql, and scale up and out. Magic. When the PG administration experience is that good I'll move it all over.
russellthehippo··on Ask HN: How do I build a website as a backend engineer?
For a backend person, this is the easiest route - code in Python, built-in or commnunity-provided javascript/css tie-ins, makes sites that are easy to build and also look good. Only issue is that it can't be served statically, so you'd need to host it on a server - but you can get super cheap hosting <$10/mo.
russellthehippo··on Ask HN: What are interesting new developments in databases related fields?
Serverless DBaaS providers and HTTP APIs are a huge new development. Mongo Atlas Serverless, Planetscale (MySQL/Vitess https://planetscale.com), Upstash (Redis, https://upstash.com) and Neon (upcoming Postgres). They add features like merge requests on DDL and make managing DBs as easy as managing Lambda functions. This reduces maintenance overhead significantly.
russellthehippo··on Bitcoin Pizza accepts Bitcoin this time
Yeah it's actually kind of funny - real HODLfolk would never spend crypto when they could spend fiat
russellthehippo··on Why Hypercard had to die (2011)
Actual link to the "wooden stake" website is here: https://web.archive.org/web/20111203222943/http://www.thomas...
russellthehippo··on Ask HN: Show me your half baked project
brain-plasma: a fast Python dictionary interface to the Plasma shared memory object store (pyarrow project)

https://github.com/russellromney/brain-plasma

I had a speed problem repeatedly loading large dataframes into memory for a Dash project - threads can’t share object updates and Redis gets slow with large objects. So I built brain-plasma. It lets you keep objects in memory, but separate from service memory - no need to read data into memory over and over. Fast access and arbitrary namespaces. It’s not perfect but it works well.

← PreviousPage 3 of 3