My understanding is that polars can get fast(er) parsing mainly/only by leveraging arrow and parquet files.
We control all our parsers and read directly into arrow memory. This differs from pandas, which utilizes pyarrow for reading parquet and then finally has to copy the arrow memory over to pandas memory.
Nonetheless, would reading a parquet file with polars be faster than reading a csv?
Also thanks for polars! Great contribution data science!
No worries. :) Most blogs on the topic I encounter in the wild make incorrect claims, I understand the confusion.
> Nonetheless, would reading a parquet file with polars be faster than reading a csv?
Yes, much faster. Please don't use the csv format for anything of a reasonable size. It is a terrible format to process and very ambiguous.