Hacker News Reader

What is your day job?

Data science consulting for business things. Most datasets < 1M rows

Those are small datasets. Also numpy != just operations on a matrix, but as mentioned above, proper vectorization can have profound speed differences for larger datasets.

They are indeed small. But most real world problems operate on small data.

And if it’s bigger you probably ought to be using a different toolset than Python altogether.

In my experience it is exceedingly rare to find a situation where you have > 1M rows of data and need to do tabular data manipulation, and it’s not coming from a managed database sort of setup where the manipulation could be better done in the database.

Reply on news.ycombinator.com