Hacker News Reader

Comment by ajross | Hacker News Reader

Or simply if the address in question has been remapped since the original free. For big 32 bit processes that's very likely. There's nothing wrong with the implementation really, it's just oversold.

Obviously the valgrind advice is great. But I don't follow your point about mmap vs. sbrk regarding fragmentation. Fragmentation is a property of memory addresses, not the syscall used to allocate them. A big linear heap allocated with sbrk will fragment just as badly as the same heap allocated in a single block, even if it's allowed to have "holes" in it.

Maybe you're claiming that jemalloc deliberatly leaves gaps between allocations? That can help a little in practice, though the real effect is just to change the size of the fragmented blocks.

View on HN

Actually also with a 64 bit address space, you can easily get the same address if the allocator returns a cached block.

The problem with sbrk() is that if you allocate a few megabytes of memory + a tiny chunk, even after freeing everything apart from the tiny chunk near the "program break", nothing will be given back to the OS. If you use mmap() instead though, all pages can be given back apart from the one where the tiny chunk resides in. This makes for a tremendous difference sometimes.

OK, I understand. The symptom you're describing isn't really "fragmentation". Fragmentation is the inability to use smaller blocks of memory because the larger allocations won't fit. That behavior isn't changed by this.

You're talking about a resource reclamation issue. Unmapping a page is a clear signal to the kernel that the memory is unused and can be repurposed immediately. Otherwise, it needs to find and eject a page from memory using the VM system, which is more expensive (though I'd guess not a lot more expensive except in pathologically allocation-bound systems).

You are right, it's not fragmentation but resource reclamation. I've added a note to the initial post.

I'm not sure though that the VM system can reclaim memory in the heap allocated with sbrk(). At least I've never seen that before. Or do you mean "eject" as in swap out?

The mmap() implementation is awesome. I used to write code for a network hardware management computer which ran OpenBSD on a memory-constrained Soekris (quasi-embedded environment more or less). The fact that OpenBSD's implementation returns memory when allocations are free()'d saved my bacon more than once. With another malloc I'd be out of memory way sooner.

OpenBSD can't swap?

It can, but this system was running on a Compact Flash (real small, I think maybe 32M usable space) so swapping was out of the question. This was also before CF got good and started mapping in good sectors when bad got overwritten.

Reply on news.ycombinator.com