Comment by catwell | Hacker News Reader

Comment by catwell | Hacker News Reader

In case someone involved in this reads this thread: the document does not specify which LMDB version you tested. I suggest you run your tests with the `mdb.master` branch, i.e. the work towards a future 1.0, and not the stable 0.9 branch. The answers to several of your interrogations will depend on that: with `mdb.master` you can use the VL32 mode which greatly improves usage on 32 bit platforms, and Windows support is much better.

Regarding NFS: I have recently started testing LMDB on NFS v4 and had no issues so far, but with a single process using the database. AFAIK the warning at http://www.lmdb.tech/doc/ is only for multiple processes using the DB concurrently. I am still not entirely sure there won't be any mmap-related issues, but so far so good.

Regarding "being careful": this is a very important point. The LMDB API does not hold your hand, it lets you do dangerous things which will result in corruption of your database, which you will discover too late. I suggest writing a wrapper around the API to ensure you are using it correctly. (I wish there was a compile flag like LUA_USE_APICHECK [1] for LMDB, which could help detect problems like this, but there isn't.)

[1] https://www.lua.org/manual/5.3/manual.html#4

The main reason for that warning about NFS is that people will try it, see that it seems to work, and then get careless and try to use the same DB from two different hosts at once. It's inevitable when you're working on files living on networked filesystems, and it cannot work. NFS doesn't offer any cache coherency guarantees, and the mutexes used for synchronizing writers only work on the host that created them.

Not sure what you're talking about re: the API letting you corrupt your database.

If we're using lmdb for something that's largely or entirely read only, how viable is using it on NFS with (shudder) lock files or something like that?

If it's 100% read-only, you could probably use it safely. Make sure its filesystem on the NFS server is mounted read-only, and obviously all the clients' NFS mounts must also be read-only. As for "largely read only" - if there are any writers at all, all bets are off.

LMDB automatically detects read-only filesystems, and turns off its locking in that case, so it should perform as well as anyone could expect NFS to perform.

Things like closing DBIs, transactions and cursors in the wrong order, for instance.

That won't corrupt the DB. It might corrupt the heap of your current process, but the DB won't be harmed.

C doesn't have destructors. If you free stuff in the wrong order, you lose. That's nothing to do with LMDB's API. Even if LMDB cleaned everything up internally, you'd still have dangling pointers in your app space. No language or API can prevent that.

The only thing you can do in the API that might possibly corrupt the DB is to use MDB_RESERVE and then store a value bigger than the space you reserved. That generally causes a SEGV, and you'll discover very quickly that you have a bug in your code. LMDB will fail fast, every time, and every failure will be a bug in your own code. Makes debugging very quick.

Glad to hear that.

To be honest the only real corruption issues I had with LMDB in practice were dues to me using MDB_NOSYNC, and broken mmap behavior on some Android devices (on external storage).

Reply on news.ycombinator.com