I'm confused by some of this. Specifically, pandas columns are essentially just a thin wrapper around record arrays in numpy right? How could it be that much faster to use numpy over pandas if it's effectively the same thing?
Right, each pandas column is a numpy array (or optionally pyarrow in pandas 2.0), but numpy is going to be faster for things that needs to be vectorized across columns, such as matrix operations.