This is probably dumb intervention but forgive me because it made me giddy to find: the pandas drop_duplicates() method is pretty great, if you're in a pinch. Again -- forgive me if you know about it!