Hacker News Reader

the csv files are not that big. a few thousand rows. But there are lots of duplicates rows I need to get rid of

Exact duplicate rows? If you have access to standard Unix tools, try this on a command line:

cat input_file.csv | sort | uniq > output_file.csv