Comment by justinsaccount | Hacker News Reader

Comment by justinsaccount | Hacker News Reader

Parent Full thread

This is impressive, but...

"A fast reader exploits the capabilities of the storage system"...

the graphs show that their storage system is doing 4.00 GB/sec

I wonder what processor this is running on and what their storage system this is.. multiple PCIe SSD?

I tried running a quick test but only succeeded in OOMing my 8G laptop.

Even just doing

  import paratext
  it = paratext.load_csv_as_iterator("/dev/shm/tmp/c.log", expand=True, forget=True)
  x = it.next()

Starts eating up all my ram after about a minute of spinning the cpu... so I think they have a slightly different definition of an iterator as everyone else.

Compared to

  cut -d , -f 5 < c.log > /dev/null

which runs in a few seconds, or a slightly more domain specific and optimized version of 'cut'[1] that runs even faster (300-500MB/s on a single core depending on which fields you want)

  $ du -hs c.log ;wc -l c.log 
  2.1G	c.log
  16197412 c.log

I also wonder if that is 2.5 GB/s per core.

https://github.com/BurntSushi/rust-csv does 241 MB/s in raw mode, so I find it a little hard to believe that this is 10x faster... unless that is while maxing out multiple cores.

[1] https://github.com/bro/bro-aux/blob/master/bro-cut/bro-cut.c

From looking at their source briefly, I can make two observations:

Firstly, their core parsing routine is a state machine not unlike the one found in rust-csv: https://github.com/wiseio/paratext/blob/master/src/csv/rowba...

Secondly, they do indeed appear to be achieving high parsing speed using coarsely grained parallelism. You can see the code that chunks up the CSV data here: https://github.com/wiseio/paratext/blob/master/src/generic/c...

It's pretty clever!

Can this be used to speed up rust-csv?

It feels like something that should be built on top of rust-csv.

I thought this was base ability of the parser to parse CSV in chunks.

Does cut work with quoted fields correctly? My understanding was that it was just a dumb line tokenizer and CSV is a little more complex than that, e.g., CSV rows can span more than one line when fields contain line breaks.

No, it doesn't and therefore is no viable option for many real-world datasets.

Which means an effective thing to do when you control the data pipeline is to ban tabs and line-breaks from your values, and then use `cut` on tab-separated files.

Sure, if you can redefine your data format, you can take shortcuts. But otherwise comparing cut to a CSV parser is just a different way of saying »if it doesn't have to be correct, I can make it as fast as I want«.

Or just not use CSV as a data transfer format :D

I think you're missing the point. Unix tools like `cut` are really effective ways to deal with plain text data, who cares if it's CSV.

> 4.00 GB/sec

> I wonder what processor this is running on and what their storage system this is.. multiple PCIe SSD?

Consumer grade NVMe disks currently achiveve 2GB/s. It is easy to use a couple of those, or just one professional SSD.

> Starts eating up all my ram after about a minute of spinning the cpu... so I think they have a slightly different definition of an iterator as everyone else.

I don't think so, the difference is you're expecting it to read one row at a time and it's actually reading one column at a time. load_csv_as_iterator is iterating over the columns.

> I wonder what processor this is running on and what their storage system this is.. multiple PCIe SSD?

One high-end PCIe SSD can manage that; two easily could. A high-end NAS might, too.

Reply on news.ycombinator.com