Hacker News Reader

Comment by gwbas1c | Hacker News Reader

How large is the SQLite database you're syncing?

Is it even "worth" using SQLite at this point? What about a configuration file, and straight-up code that works with in-memory data structures?

View on HN

This is something we seriously considered. The SQLite dbs are several hundred megabytes in size (millions of IP ranges) so while it would be technically doable to send around rules files as JSON or something more specifically suited there's still a number of wins that SQLite gives us:

- Really strong support across multiple platforms (we have clients for most of the major web frameworks)

- Efficiency, sure we have lots of RAM on servers nowdays but on some platforms it's constrained and if you don't have to burn it, we'd just rather not.

- When we started mapping this out, we ended up with something that looked like a JSON format that we were adding indexes to....and then we were re-inventing SQLite.

I don't know how it works exactly, but I believe you can have a fully in-memory SQLite database. Bun's sqlite library and SqlAlchemy both let you operate on in-memory SQLite db's which you can then write to disk.

Edit: reading the docs it looks like it operates the same way, just reading sections of the db from memory instead of disk

https://www.sqlite.org/atomiccommit.html

You can, but that's not the point. I basically asked if they should (gasp) write code that did the lookup.

See the other response from the article's author.

Reply on news.ycombinator.com