I don't understand this reasoning. Why is it being limited to main memory speed? Surely the yes program, the fragments of the OS being used, and the program reading the data, all fit within the L2 cache?
For each byte of data passing through pv:
1. the byte is read from yes memory to CPU registers by the write syscall handler in the kernel
2. the byte is written to kernel's internal buffer associated with the pipe
3. the byte is read back in the read syscall called by pv
4. the byte is written to a buffer in pv memory
5. and thas's the end because write syscall executed by pv on /dev/null very likely doesn't actually bother reading the submitted buffer at all
edit: Actually it might only be 20GB/s because on Linux pv seems to use the splice syscall to transfer data from stdin to stdout without copying to userspace.
This is also the reason why further "optimization" of this program in assembly was a fool's errand: the bulk of CPU load is in the kernel.
The actual throughput then, once you include OS copying, is either 2 or 4 times the quoted speed (depending on splice usage), so we're either at main memory theoretical speeds, or double main memory speeds. Intuitively, I'd still have expected that it should be a larger multiple.
(A quick search can't find me any reliable Intel L1/L2 cache speeds/multiples to quote, so I admit this comment is more speculation than it should be!)
See eg. https://stackoverflow.com/questions/23599074/system-calls-ov... who benchmarked it at ~638ns per "read" call.
(Many, many years ago I was working on the Zeus web server, and we went to surprising lengths to avoid syscalls for performance.)
A read() syscall takes longer than a getpid() syscall because read() has more work to do, it actually does a data copy of len bytes, which takes some time (and will be faster/slower if data is cache hot)
What we call the "syscall overhead" is what happens before and after the actual data copy, switching between user and kernel mode.
You make that overhead negligible by calling read() with a large size.
Snap!
IIRC, ZWS used shared memory to mirror the results of the time() syscall across processes, to save a few nanoseconds on some operating systems :) That was before Linux and other OSs used techniques like the vsyscall/VDSO mentioned in the stackoverflow discussion...
$ yes |pv > /dev/null
46.6GiB 0:00:05 [9.33GiB/s]
$ taskset 1 yes |taskset 1 pv > /dev/null
32.9GiB 0:00:05 [6.58GiB/s]
$ taskset 1 yes |taskset 2 pv > /dev/null
45.7GiB 0:00:05 [9.13GiB/s]
$ taskset 1 yes |taskset 4 pv > /dev/null
45.7GiB 0:00:05 [9.18GiB/s]
I tried 2/4 as not sure how 'real' cores vs 'hyperthread' cores are numbered. These numbers are from a i7-7700k.
Assuming pv uses splice(), there is one only copy in the workload: copy_from_user() from fixed source buffer to some kernel allocated page, then those pages are spliced to /dev/null.
If the pages are not "recycled" (through LRU scheme for allocation), the destination changes every time and the L2 cache is constantly trashed.
https://www.reddit.com/r/unix/comments/6gxduc/how_is_gnu_yes...
There's no reason that the L2 cache needed to be flushed at any point - the caches are all dealing with physical memory, rather than virtualised address space, so the fact that there are two processes here shouldn't stop the caching from working.
It's completely possible for the data to not (all) leave the CPU. If the caches are large enough, then the pages full of "y\n" will be still resident in the cache when the next iteration of the program overwrites the same pages again. Then the CPU has no need to send the original page out to main memory.