Hacker News Reader

Comment by necubi | Hacker News Reader

Totally different use-cases. This is an embedded key value store, not an RDBMS. You would use this in place of e.g., LevelDB or RocksDB, potentially as the storage layer of a database.

View on HN

There's always the venerable:

  CREATE TABLE kv (
    k TEXT PRIMARY KEY,
    v TEXT NOT NULL
  );

Even if sqlite is technically an RDBMS, I think it's a legitimate comparison. Is SplinterDB worth giving up sqlite's reliability and feature set?

This is much lower-level than sqlite. In fact, you could use this as the storage layer for a SQL DB. See, e.g., MyRocks[0] which is a MySQL backend that uses RocksDB as the storage layer.

In other words, you'd use this when you just need a persistent KV store and want to build the higher level semantics according to your application's needs.

[0] http://myrocks.io/

> In other words, you'd use this when you just need a persistent KV store and want to build the higher level semantics according to your application's needs.

Why can't you use SQLite for this usecase? I believe FDB uses SQLite as an embedded KV store.

What it comes down to is performance.

You can use a relational database such as SQLite for a low-level key-value store, such as RocksDB or SplinterDB, but then you pay for the higher-level semantics with lower performance.

I'm not very familiar with foundationdb, but I'm confident they're not using sqlite as the storage layer. That would come with a tremendous performance penalty. The docs say that "The SSD storage engine stores the data in a B-tree based on SQLite" which makes me think that they're just using the storage layer from SQLite (i.e., the part of it that corresponds to splinterdb/rocksdb).

Your confidence is misplaced. The storage layer is SQLite.

They are leaving sqlite

In the upcoming FoundationDB 7.0 release, the B-tree storage engine will be replaced with a brand new Redwood engine.

https://apple.github.io/foundationdb/architecture.html#stora...

I did see a full copy of the SQLite amalgamation file in the FDB codebase, but you're probably right that they might be using internal APIs.

I'm still skeptical of the "tremendous performance penalty" you'd suffer from using SQLite. Just because you do fewer things doesn't necessarily mean you're faster at doing them. I've hit ~120,000 inserts/sec on SQLite without weakening any of it's durability guarantees. If you play fast and loose with fsync and WAL, I'm sure you can squeeze out even more performance.

I can also think of use-cases where you don't want the write amplification that comes with RocksDB or the memory constraints of LMDB.

I am not familiar with SplinterDB, but I do have a lot of familiarity with RocksDB. These types of k/v storage layers are designed to handle hundreds of thousands of not millions of operations per sec. Especially the way they handle writes (typically with an LSM) is very different from SQLite, and it shows in terms of throughput of e.g. random writes.

I'd say that these low-level storage engines have more in common with filesystems than SQLite, they're just not in the same ballpark at all.

You mean like dict() in Python? What's the use case for this?

It's like a dict, but persisted to disk. This means that it's both durable (so if your process/machine crashes, you don't lose data) and also can store datasets larger than memory.

Reply on news.ycombinator.com