Hacker News Reader

Comment by exacube | Hacker News Reader

I'd love some further explanation here:

what does locking memory mean in this context? For what purpose do security sensitive programs lock their memory? what is the performance degradation that happens with low-spec computers?

View on HN

> what does locking memory mean in this context?

I think the person is referring to the mlock() and mlockall() functions (or equivalents on other OS), which keep pages resident / prevents pages from being paged out. Forces them to remain in RAM.

It can be used to, e.g., prevent an encryption key or password from being swapped out to disk, where it might then be recoverable. (Personally, this is why I encrypt swap.)

> what is the performance degradation that happens with low-spec computers?

Locking a larger portion of RAM means less room for the OS to page out unused pages and free up the space for other programs.

While one can try to selectively lock buffers with sensitive data with mlock(), you have to be sure they aren't copied into other buffers that aren't locked (and could thus be subsequently paged out). If you're writing a UI program that displays or receives those in a widget, this might be harder (you might not have access to the internal buffer of the widget, as it is an "implementation detail" of your library), and locking the entire process might be a simpler solution (albeit being a bigger hammer).

My understanding was that mlock doesn't really keep the page from being paged out, since it can't even begin to do that in the hibernation case, or if you're running as a VM who's guest RAM wasn't mlocked.

What it does is keep a canonical version of the page in memory. That's useful being able to deterministically touch a piece of memory, but it doesn't really help you as far as making sure the page never touches disk.

I hadn't considered hibernation, and indeed, a deeper reading of the manual confirms that hibernation doesn't count, which is rather interesting (given the implications of hitting disk), but I don't really see a good way around it, short of aborting the hibernation, or providing a mechanism to inform the program that those pages were lost. The man page (later, annoyingly, after its initial description) notes this:

> Memory locking has two main applications: real-time algorithms and high-security data processing. Real-time applications require deterministic timing, and, like scheduling, paging is one major cause of unexpected program execution delays. Real-time applications will usually also switch to a real-time scheduler with sched_setscheduler(2). Cryptographic security software often handles critical bytes like passwords or secret keys as data structures. As a result of paging, these secrets could be transferred onto a persistent swap store medium, where they might be accessible to the enemy long after the security software has erased the secrets in RAM and terminated. (But be aware that the suspend mode on laptops and some desktop computers will save a copy of the system's RAM to disk, regardless of memory locks.)

I would be rather disappointed if a hypervisor swapped out my guest (at least, in a context like AWS; I suppose if you're just running qemu on your laptop, that's a different matter), but I hadn't considered that either, and it is certainly possible.

IMO, the right answer is to better define your threat model. Are you concerned about someone pulling the HDD and reading the swap? Use an FDE scheme that covers your swap too. Are you concerned about someone getting access to the swap file programmatically? At that point they have so many other ways of slurping the memory out of your process that it's a lost cause.

Good explanations of what and why one might lock memory.

But, in the context of this overall discussion, I think it's important to keep in mind that when a process allocates a large amount of virtual memory, it does not automatically allocate any physical memory. So a process allocating a lot of virtual memory up front should not impact other processes which have locked some of their memory into physical memory.

> I think it's important to keep in mind that when a process allocates a large amount of virtual memory, it does not automatically allocate any physical memory.

It will, if you mlock() it, I believe. The manual page notes "real-time processes" as a main user of mlock() (the other being the cryptographic uses I hinted at); it cites their use case as locking the page to avoid delays due to paging during critical sections. In order for that to work, the OS would need to bring the pages in, at the time of locking; so at that point, a large virtual allocation becomes equivalent to a physical one.

Correct, but my point was that a process with allocates a lot of physical memory will not interfere with the other processes that have locked pages.

Linux cgroups and Solaris containers atleast also provide a way to avoid paging to disk, without modifying the program.

It tells the OS to never swap memory allocated to the process to disk.

You should always lock memory if you're going to be storing crypto keys, etc. since once the pages are swapped to disk you're vulnerable to someone pulling the swap partition out and reading it.

It is significant to note that:

1. You don't have to lock all your memory (although it may be hard to capture all the intermediate buffers if you don't).

2. You still need to clear the memory buffers after they're no longer going to be used, otherwise other processes can read /proc/kcore, etc (or cool your RAM and extract it and put it another system)

3. It is possible to encrypt the swap partition with a randomly generated key at boot

One locks memory to keep it from being swapped to disk. You can imagine if you have sensitive data, you want to keep it as ephemeral as possible. Do even security-focused go programs typically lock _all_ of their virtual memory, or just the sensitive pages? I don’t know. But if a bunch of programs hogged hundreds of megabytes of unused RAM each, that’s the problem alluded to on low resource systems.

Reply on news.ycombinator.com