Hacker News Reader

Comment by eridius | Hacker News Reader

Mike Ash covered the idea of using virtual memory tricks to handle ring buffers 3 years ago, written up as https://mikeash.com/pyblog/friday-qa-2012-02-03-ring-buffers....

Unlike the OP's article, Mike Ash's version uses vm_remap() to remap memory around instead of hitting the filesystem and relying on tempfs to keep the data in-memory. vm_remap() is an OS X API, and I don't know offhand if there is any equivalent on Linux (though I would be surprised if there isn't some way to do the same thing).

View on HN

ryg too wrote about "magic ring buffers" 3 years ago: https://fgiesen.wordpress.com/2012/07/21/the-magic-ring-buff...

mmap doesn't necessarily have to hit the filesystem, as there are several ways to create file descriptors that are backed by memory (POSIX shared memory, typed memory objects, for instance).

Is this the general idea?

    int fd = shm_open("test", O_RDWR|O_CREAT, 0600);
    ftruncate(fd, 4096*2);

    char *part1 = mmap(NULL, 4096, PROT_WRITE|PROT_WRITE, MAP_SHARED, fd, 0);
    char *part2 = mmap(part1 + 4096, 4096, PROT_WRITE|PROT_WRITE, MAP_SHARED|MAP_FIXED, fd, 0);

    part1[0] = 'X';

    assert(part2 - part1 == 4096);
    assert(part1[0] == part2[0]);

    shm_unlink("test");

I've programmed with POSIX shared memory, and yes, that's the general idea.

A very important caveat is that the POSIX shared memory namespace is shared among all processes, so you need to wrap shm_open with a mkstemp()-style algorithm that generates a random name, opens with O_EXCL, and tries again if it fails. Unfortunately it's very easy to mess that up and introduce a security vulnerability.

It would be also prefferable to unlink just after /open/ call.

Wouldn't mkstemp("/dev/shm/tmp-XXXXXX") be enough. As strace tells /shm_open/ is just a wrapper for /open/ with /dev/shm/ prefix.

On Linux, shm_open is implemented with /dev/shm, but that's not the case on other platforms.

Presumably you have to worry about file descriptor limits even if your fd doesn't hit the filesystem. vm_remap in OS X is nice because it doesn't need file descriptors at all.

POSIX shared memory is implemented on linux systems as a mmap'd tmpfs file...

Best way is to use mmap MAP_ANONYMOUS. Or with kernels 3.17+ memfd.

The funny thing is, I was asking about a function like memfd over 10 years ago:

https://groups.google.com/forum/#!topic/comp.os.linux.develo...

Reply on news.ycombinator.com