You cannot parallelize csv parsing by chunking like this. The format prohibits it, full stop. This is why xsv comes with an `xsv index` command for example.
Like I said, either I'm misunderstanding the optimization they are doing or they aren't being transparent. From a glance at the code, it looks like the latter to me.
(As others have mentioned, you can parallelize csv parsing, but it requires multiple passes. Which is basically what `xsv index` is. It lets you choose when to do the first pass, and then benefit in performance in all subsequent runs.)
Use a fast "frontier" thread to determine quotes and split works at the safe boundary.
I'm personally okay with the speculative execution as long as its result is correct and I wouldn't consider it cheating. But it is concerning that the benchmark doesn't list all the caveats (as all proper benchmarks should do, like [1], which is incidentally written by who you are replying to). I can't even determine whether the machine is hitting I/O bound or not from the OP!
Doing two pass parsing: https://liuliu.me/eyes/loading-csv-file-at-the-speed-limit-o...