Hacker News Reader

Comment by inmemorydotnet | Hacker News Reader

We've an InMemory Dot Net database, and we've seen cases of hour long GC pauses. Typically this happens on 64 bit systems, that are low on memory, and have needed to start to swap to disk. The swap file, being located on Network storage doesn't help.

View on HN

My default action to ensure good service quality for GC'd systems is to, as far as possible, minimize what swapping can occur. Especially important for systems with a big heap containing many rarely used objects taking up a substantial fraction of the space, more so if GC pressure is very low.

This is regime where GC languages simply can't coexist with aggressive swapping. You have to either disable swapping (somehow), or use something like a shared library doing something like the Linux mlockall for the runtime heap.

I believe there is something similar to mlockall you can do on windows to somewhat prevent swapping out the heap, but unfortunately I can't recall it right now.

I remember a research paper about a GC which worked well even with swapping but it required hooks in the kernel memory manager and the researchers weren't able to make their patch of linux accepted.

Why on earth would you configure your swap file on a network path?!

I've seen virtualized environments where every single scrap of disk is located on a SAN, because that's easier/cheaper/something-er for IT to manage.

While technically a SAN is networked storage, I read the OP's comment to indicate something more like a SMB share.

Our enterprise databases store the data on high-performance SAN devices, and these are not slow. We moved from local spinning-rust drives to SAN SSDs, and our performance improved.

Why on earth would you configure swap on a system running an in-memory database? Presumably you use an in-memory database because you need extremely fast response times from it, but those are going to go through the roof when it starts swapping, potentially causing a cascading failure throughout systems that directly or indirectly rely on it.

Better to configure the databse to allocate less than the available system memory and refuse writes when it's out of free memory.

Don’t worry, the in-use parts of the swap file will be cached in memory

Windows requires some swap to work correctly. Unlike Linux, it doesn't ever overcommit. IIRC, too little swap can make some parts of Windows work incorrectly.

Do you have an example of which "parts of Windows work incorrectly"? Windows runs just fine without a pagefile. Of course if you have a piece of software which then tries to use more memory than is physically available those allocations will fail, which in 99.99% of software results in termination or a crash, but the OS itself works just fine (as does all the software as long as you're not hitting your physical memory limits).

I don't remember exactly what it was, but I do remember you needed about 200-300MB page file to avoid it.

Of course, due to Windows no-overcommit policy, it's better to have a large page file almost always. That ensures your RAM is going to be used efficiently and not as a simple backing store that's never actually referenced.

Reply on news.ycombinator.com