Discovering and exploring mmap using Go | Hacker News Reader

Discovering and exploring mmap using Go

brunocalza.me

107 pointsbrunoac10 comments

For people doing memory mapping in Go, I would strongly advise calling debug.SetPanicOnFault() and setting up a panic handler. Without it, your program will simply crash in case of I/O failures, file truncation, etc..

Here's some code I wrote some time ago that does exactly this:

https://github.com/buildbarn/bb-storage/blob/c346ca331930f1b...

Be aware that using mmap instead of read/write might mess up Go’s scheduling. The runtime has special handling of other blocking system calls which tried to minimize the impact of those on other running goroutines. With directly accessing memory locations mapped via mmap you won’t get this benefit.

This kind of articles, sounds like a reminder that system programming is a fundamental knowledge for software engineers.

Fundamentals are key. And it is very fun for those who really enjoy their craft

Does this assume the file fits in address space? That's not always the case.

While the default behavior of this library is to map the entire file into address space (and also the default assumption that people have regarding mmap), you can map specific portions of a file into memory to avoid virtual address exhaustion. It's just far less convenient to do so, though typically still faster than the alternatives.

From the first paragraph:

> One of the main problems a database storage engine has to solve is how to deal with data in disk that is bigger than the available memory.

I misread the comment. The file fits in address space? As sibling comment suggests, address space is v. big and it's very much a corner case (or very specific use case) for most people.

All x86-64 CPUs support 48 bit address space, with 57 bits in the newest ones [1]. It's unlikely that address space exhaustion would be a problem for the vast majority of use cases.

[1] - https://en.wikipedia.org/wiki/Intel_5-level_paging

According to the developers of Mongo at least, it's not worth considering. You can't store more than 2GB of data on a 32-bit machine because they map the entire datastore into memory.

Reply on news.ycombinator.com