Hacker News Reader

Hacker News

Comment by joshlk | Hacker News Reader

Full thread

joshlk

The Pandas CSV reader (Python) is one of the best and most versatile implementations out there. Also very fast.

https://pandas.pydata.org/pandas-docs/stable/reference/api/p...

View on HN

Also xsv [0] and the related Rust csv crate [1]. xsv in particular is the best CLI csv tool I've ever used.

And I've used them all.

[0] https://github.com/BurntSushi/xsv

[1] https://github.com/BurntSushi/rust-csv

Wow! That's a nifty little library. Just looking at the amount of options going into the main parser routine is mind blowing [1].

And the figuring out of encoding is also complex [2].

Once again, this is just to demonstrate that writing your own CSV parser from scratch is a total waste of time. Just use tools provided by your language. Many languages provide native support for CSV parsing. For example see docs for Microsoft C# [3].

----------

[1] https://github.com/pandas-dev/pandas/blob/v1.0.3/pandas/io/p...

[2] https://github.com/pandas-dev/pandas/blob/v1.0.3/pandas/io/p...

[3] https://docs.microsoft.com/en-us/dotnet/csharp/programming-g...

string.Split is far from native support.

If you don’t want to use third-party libraries then TextFieldParser[0] is part of the .NET framework.

I use CsvHelper[1] in my projects. It can do pretty much anything although the newer versions have some dependencies.

[0] https://docs.microsoft.com/en-us/dotnet/api/microsoft.visual...

[1] https://joshclose.github.io/CsvHelper/

Disagree.

Data.table’s fread is leagues ahead of pandas.

https://www.rdocumentation.org/packages/data.table/versions/...

Fread has automatic footer detection and has automatic skip logic to help parse out mangled headers in some csvs.

fread can also accept a shell command which it will run and use the result as input. I use that to grep the lines I want from files too big for RAM.

Same here!

I also use fwrite to send it back to the shell to continue my processing pipeline.

https://www.rdocumentation.org/packages/data.table/versions/...

It's mind boggling how little fanfare data.table has despite it being the best way to handle tabular data in 2020.

fread is amazing. Whenever I look at the docs and see all the options for dealing with nested lists (like the within-column-seperator) I always pray never to have need of such a thing.

Additionally, RStudio has a built in GUI widget for importing .csv (and other) files: https://support.rstudio.com/hc/en-us/articles/218611977-Impo.... I find it quite useful for breezing through the common gotchas with importing data, and it’s probably the easiest way for inexperienced programmers (e.g. business analyst types used to working in Excel) to work with .csv.

In what ways is it better than the csv module in the Python Standard Library? Or in other words, in which cases is it worth the effort to install pandas just for reading CSV files if your project otherwise doesn't use pandas?

I wrote a Python utility to help our folks at our analytics company load/unload/move data among CSV files and SQL databases.

Getting the parts that work with CSVs to "just work" when presented with an arbitrary CSV file has been super interesting - even with Pandas at our disposal.

https://github.com/bluelabsio/records-mover/blob/d18ec02fdf5...

I wrote up some of the experience here:

https://github.com/bluelabsio/records-mover/blob/master/docs...

Are we talking "fast", or merely "fast for Python"?

It sounds like you may not consider anything python can do to be fast, so I'm thinking the latter.

The built-in

    import csv

never failed to do everything I needed, what does this Panda thing add?

In Rubyland there's the versatile CsvReader library / gem [1] that supports HXL, ARFF, CVY and more "exotic" or modern flavors / dialects among many others.

[1]: https://github.com/csvreader/csvreader

N.B. Floats won't rountdrip by default (you need to call `read_csv(..., float_format="round_trip")`)

Reply on news.ycombinator.com