Hacker News Reader

Comment by floating-io | Hacker News Reader

The seemingly critical parallel job performance issue sounds like it can and should be handled transparently by the OS when multiple pages are involved, and should thus be an area of research if it isn't already.

Frankly, the idea that it isn't already a solved problem makes me wonder if that could possibly be the case...

The writes thing... Outside of very rare circumstances, I would think that properly batching inserts/updates within transactions at the app level would resolve half of that, and multi-user concurrency would buy back the rest.

What am I missing?

P.S. I'm mad at the author for enlightening me to the real-world performance impact of enterprise SSD. i'm about to buy, and that might just cost me...

View on HN

> The seemingly critical parallel job performance issue sounds like it can and should be handled transparently by the OS when multiple pages are involved,

A thread can only fault on one page at a time, and only sequential access patterns can benefit from the OS bringing in multiple pages per fault. Having all of your CPU cores busy handling page faults and context switches to threads that are about to trigger another page fault is only going to generate enough read traffic to keep a few SSDs busy, not a full shelf of 24+ drives. mmap() and synchronous IO don't scale well enough for today's SSDs.

That sounds like it would be an issue for anything. If the kernel knows that N pages need to be paged in, it should be able to optimize that behind the scenes in whatever way is needed.

I'll agree that mmap() would probably have issues with prediction there (other than basic readahead), but I would think that issuing a simple bulk read() call (or more likely an equivalent zero copy call; vread() if memory serves? Been too long) would give the kernel enough information to optimize bulk transfers.

Small async reads will always be slow if done serially, but there's already "standard" scatter/gather calls that should let the kernel optimize that as well.

In short, all but a truly naive application should be limited by the SSD or PCIe bandwidth before they ever hit issues like that if the kernel is optimizing correctly, and assuming that the application can predict its own needs.

Reply on news.ycombinator.com