Comment by vitiral | Hacker News Reader

Comment by vitiral | Hacker News Reader

Parent Full thread

I question the methodology.

To measure this I would have N processes reading the file from disk with the max number of parallel heads (typically 16 I think). These would go straight into memory. It's possible you could do this with one process and the kernel will split up the block read into 16 parallel reads as well, needs investigation.

Then I would use the rest of the compute for number crunching as fast as possible using as many available cores as possible: for this problem, I think that would basically boil down to a map reduce. Possibly a lock-free concurrent hashmap could be competitive.

Now, run these in parallel and measure the real time from start to finish of both. Also gets the total CPU time spent for reference.

I'm pretty sure the author's results are polluted: while they are processing data the kernel is caching the next block. Also, it's not really fair to compare single threaded disk IO to a single process: one of the reasons for IO being a bottleneck is that it has concurrency constraints. Never the less I would be interested in both the single threaded and concurrent results.

I agree, I think there is often a faulty assumption by many developers doing benchmarking that their underlying environment doesn't matter. Often I see performance results posted with little mention of the details of the environment. At least here they posted they were using a high-end SSD, but note they just say they're on a "Dell XPS 13", as if there aren't multiple variants of that model produced every year for the last 5 or 6 years.

You're probably also right their multiple test runs resulted in the OS caching data, and a lot of the test runs may have just been testing in-memory performance instead of raw storage I/O performance.

Fair call about "Dell XPS 13". I've now clarified in the article (it's a recent 2022 Dell XPS 13 Plus).

Regarding OS caching: I'm trying to avoid this by clearing caches with the "sysctl vm.drop_caches=3" command. Note that I show both cached and uncached numbers.

It would still be good to call out the exact processor model, as the lowest end i5-1240P has half the L3 cache as the highest i7-1280P.

Also I don't think it was clear you were running "sysctl vm.drop_caches=3" between benchmarking runs of your optimizations. Your table seemed to indicate those were generic initial read/write benchmarks from either dd or hardparm. The site you linked to also had comments on it stating dd is not very good for benchmarking, suggesting fio & a different site[1].

1. https://linuxreviews.org/HOWTO_Test_Disk_I/O_Performance

Thank you. I'd be pretty annoyed in this interview. Surely my potential employer would be more interested in having me apply my twenty years of real-world experiences to what I learned in CS240.

Not sure why you'd be annoyed? What I'm trying to gauge in my interviews is their real-world experience (not their academic CS knowledge). That's part of the reason I find my line of questioning helpful: I don't penalize them for a "wrong" answer, but more how they think about it. Then we can discuss parallelization, map-reduce techniques, profiling and measurement, and so on.

Annoyed because of the framing, that is: the answer isn't "wrong" in the first place.

Interviews usually have extremely limited time, especially if some interesting “tangent” comes up, which often tell the true depth of their knowledge. Maybe the concurrency would be one possible tangent.

Regardless, the concurrent approach would be 90% the same as the single thread approach, leaving it for a good “after the fact” question, assuming the candidate still has time.

Reply on news.ycombinator.com