Hacker News Reader

Comment by matheusmoreira | Hacker News Reader

> The whole discourse was poisoned years ago when the author of Redis told everyone to disable THP on Linux

What's the story behind that? Do normal programs really get affected by this?

Normal programs are most likely using glibc's memory allocator, and I'd be surprised if it was incapable of handling arbitrary page sizes. Only reason why I have to care is I implemented my own memory allocator.

View on HN

Redis uses jemalloc by default, if I recall correctly, but its hostility to the way systems actually work arises from the way that it forks, then changes one bit on every page in the entire virtual space, which causes a lot of kernel work to support copy-on-write by blowing up huge pages into smaller pages. That's what happens when your program is antagonistic to the way the machine actually works.

> then changes one bit on every page in the entire virtual space

Yeah that sucks. Naively implemented garbage collectors have the same problem: they put the live and mark bits in the object itself which spreads those bits all over the address space. This leads to the garbage collector touching every single page when it scans and writes all of those bits.

The proper solution is to allocate separate bitmap pages. This dramatically improves cache efficiency. Machines always want a structure of arrays.

I've always struggled a bit with the fact that "machines want SoA but readability/clarity/etc is easier with AoS". And wonder if it would be possible for a language to have the code representation be AoS but the implementation transparently be SoA.

Pretty old, but a good example of how such an abstraction works: https://github.com/ExaScience/arrow-street

Thanks for explaining, I can see how it got to where it is.

Reply on news.ycombinator.com