The MySQL “swap insanity” problem and the effects of the NUMA architecture

Use rsync. It now preserves the buffer cache status that files had before so it does not stomp on your allocations.

http://insights.oetiker.ch/linux/fadvise/

I suspect the reason to be that the system has noticed that your apps haven't touched their memory for a long time and thus "don't need it". This is a reasonably valid assumption on servers: if you've got some daemons that are backgrounded for minutes or hours at a time, keeping their memory resident is a waste. However, on the desktop responsiveness (latency) is more important than throughput. Just because you only switch between apps on a timescale of minutes or hours, doesn't mean the kernel should swap them out. So the algorithm needs different weighting.

I used to experience this problem, but I haven't lately. I suspect what's going on is that the "-desktop" kernel variant of OpenSUSE uses a differently weighted swappiness algorithm. If your distro offers a choice of different kernel variants, you could try them; otherwise (or if that doesn't help), you could track down the knobs you need to tweak to make the problem go away.

I guess that the kernel cannot know that the application (in your case scp) will not try to touch the data ever again.

The current default behavior, which you call crazy, does however favor programs which do actually need the data that was just read (e.g., databases).

Would mmapping those files instead of reading them provide saner behavior, or does the kernel still do that?

I'm guessing you're either running a rather old distribution/kernel, or your swappiness is set far too aggressively. Try setting it to 0.

The problem is that if you disable swap completely and let the system refuse allocation, you will face failures not just from the process that was responsible for taking most of the memory, but from existing ones that attempt to continue operating normally -- including innocuous processes such as bash and sshd.

This would affect C programs in particular, since they usually manage their memory manually. If bash can't malloc() a buffer for its input, for example, it will simply fail, and you might be able to do anything to fix the system; the same goes for sshd, which might end up refusing new connections as a result. Programs that preallocate important data structures, and programs using garbage collection, would fare somewhat better.

In other words, if swap is disabled you will still need a sort of soft limit or reserved space to ensure that programs can survive memory starvation. I don't know if the Linux kernel (or the GNU C library) has anything of the sort.

Well, for the systems in question, they don't necessarily need swap, and they may never actually swap in practice aside from problems such as the NUMA issues described. However, even with swap disabled, the NUMA issues cause problems with performance due to running a single node out of memory.

Because most data is critical and you can't afford to just drop it on the ground whenever you please. A better option would be to have the application/db to have its own swap routines optimized for its own purposes rather than letting the OS doing a catch-all swap method.

You don't need it on some systems, however Linux will typically kernel panic if it has no swap rather than just killing processes. FreeBSD/NetBSD/Solaris all do this properly, however its still good to have swap because having your processes die when you make a tiny mistake is rather irritating. This is a great example of why you need experienced ops people to reign in the ideas of green engineers.

Swapping made sense when we had computers with limited resources running multi-user workloads. Neither is now true. These servers are special purposed so they don't need the overhead of a general OS at all. But aren't there two issues here? Swap and node allocation? Even without swapping wouldn't you have NUMA node allocation issues?

Thanks! I am glad you learned something, and happy to get great feedback!

They have not changed in any way, however there is a patchset proposed currently to change how NUMA works a bit. Unclear if it will change this situation.

Pretty hard to make you happy, eh?

I agree with you on a purely technical basis, however, this article was written for the MySQL community, was tested (only) on MySQL, and has primarily affected my only on MySQL systems, which I (and the others referenced in the article) primarily run on Linux.

Thats a very good point, these types of problems exist to some extent on other OSes but this case seems pretty specific to Linux. I suspect testing on Solaris/FreeBSD would show better results in this area.

This applies to any case where you want a single process to use more than half the server's RAM.

Yes, absolutely. In fact one of the most common longer term referrers for that post is about MongoDB, not MySQL:

http://www.mongodb.org/display/DOCS/NUMA

Except that's not the TL;DR at all.

From the article:

"An aside on zone_reclaim_mode

The zone_reclaim_mode tunable in /proc/sys/vm can be used to fine-tune memory reclamation policies in a NUMA system. Subject to some clarifications from the linux-mm mailing list, it doesn’t seem to help in this case."

The real TL;DR is "run your mysql command under the auspices of '/usr/bin/numactl --interleave all' so that your big pool allocation is split evenly across nodes"

And an even better solution would be if _only_ the big pool allocation use interleaved allocation, and all the rest used normal node-bound allocation. This would require some sort of change to the malloc calls though, yes? All of the solutions listed in the article operate at the granularity of a process (or higher), not down to the individual allocation.