Here's an example using csvkit:

    $ time xsv stats --everything /tmp/nfl_all_plays.csv > stats.csv
    real    5.723
    user    14.390
    sys     1.914
    
    $ time csvstat /tmp/nfl_all_plays.csv
    ^C after 2.5 minutes

Here's the data: https://burntsushi.net/stuff/nfl_all_plays.csv

It's only 74MB. Let's take a smaller slice to see how long csvstat really takes. This is just the first 10,000 records, which is ~3MB: https://burntsushi.net/stuff/nfl_all_plays_small.csv

    $ time csvstat /tmp/nfl_all_plays_small.csv > /tmp/stats.csv
    
    real    1:01.85
    user    1:01.70
    sys     0.103
    
    $ time xsv stats --everything /tmp/nfl_all_plays_small.csv > /tmp/stats.csv
    
    real    0.308
    user    0.576
    sys     0.071

Now technically, csvstat is doing more work in that it seems to be computing a frequency table as well. But we can just do the same for xsv and add the time, with the knowledge that it would be faster if it were coupled into `xsv stats`:

    $ time xsv frequency /tmp/nfl_all_plays_small.csv > /tmp/frequency.csv
    real    0.251
    user    0.187
    sys     0.063

Now let's see how xsv fairs on a much larger sample, which is just nfl_all_plays.csv repeated 10 times and is ~800MB:

    $ ls -lh /tmp/nfl_all_plays_huge.csv
    -rw-r--r-- 1 andrew users 806M Sep  8 20:34 /tmp/nfl_all_plays_huge.csv
    
    $ time xsv index /tmp/nfl_all_plays_huge.csv
    
    real    2.041
    user    1.876
    sys     0.163
    
    $ time xsv stats --everything /tmp/nfl_all_plays_huge.csv > /tmp/stats.csv
    
    real    28.336
    user    4:36.45
    sys     24.212
    
    $ time xsv frequency /tmp/nfl_all_plays_huge.csv > /tmp/frequency.csv
    
    real    6.077
    user    1:16.51
    sys     1.873

That indexing step lets xsv do its processing in parallel. Good luck doing that in Python without blowing your memory budget. :-) csvkit would either take hours to handle that much data or would more likely run out of memory.

With that said, I was able to write a Python program that just counted records within an order of magnitude of `xsv count`, but it was still a few times slower.