Hacker News Reader

Comment by z92 | Hacker News Reader

I was reading somewhere that Mongo can't do document level/record level locking because of mmap'ed files. The whole database is memory mapped. And mmap doesn't understand underlined data structure, it views the whole file as a large single blob.

Ditching mmap will not be that easy, cause most of the speed and simplicity of Mongo comes from using mmap.

View on HN

mmap does complicate things. A traditional database often works using write ahead logs. The log holds changes made to the database over time - so when you want to write a change to your DB, you put 'I'm changing value x.y to 50' in your WAL. Sometime later the actual data pages holding the modified x data structure can be written out to disk. If you have a crash before the data pages get written out, you can 'replay' your WAL file to redo all the lost changes.

Unfortunately, a central requirement of a write ahead log is that the data pages must not get written before the WAL. If that were to occur, and there were a crash, the system wouldn't know that it had to undo the changes to the data pages, leaving you with corrupt data. mmap generally doesn't provide the ability to pin your dirty pages in memory - they're subject to getting flushed any time the system is under memory pressure - so it makes logging a lot more complicated to get right.

Just to note, the MongoDB journal is basically a write ahead log and has been around since 1.8. It's not simple, you are correct, and involved remapping portions of memory privately, and leads to some inflated numbers on the virtual memory reporting side. There's a great write up here:

http://blog.mongodb.org/post/33700094220/how-mongodbs-journa...

That's interesting, thanks!

Yes, and an important limitation stems from this as well.

For a tool like Redis, it's fairly easy for me to accept the limitation that my data size can't exceed available RAM.

But for an indexed document store with full-text query capabilities, it's a lot harder to for me to accept that limitation.

Are you implying that a performant Mongo DB must keep all data in RAM? That's not true. We've got almost 1TB in a Mongo DB, and we sure don't have that much RAM.

You do have to be able to keep your indexes in RAM, but that's much less limiting.

Not 1:1, no - thanks for clarifying that. But depending on the content and the indexing, there is a strong correlation between the size of the database and the memory requirements under Mongo DB.

It doesn't use mmap for locking though - and in any case one database is already multiple files (2GB max). There are internal data structures used by MongoDB and the database itself takes care of locking, mmap just gets the data into memory.

Reply on news.ycombinator.com