At the point when it was loading csv file with data. I think there was 5-6GB. My machine has 32GB of ram, and when usage of the python process alone increased to 35GB, I just stopped that. I've tried a couple of times, with similar results.

I have loaded that csv file to postgres, the database had similar size. With indexing it was 15GB on disk. All queries were quite fast.

So instead of loading the data to pandas, and making searches, I just wrote some SQL, and got the same results. However much faster, and with much smaller memory usage.