The catch, IMO, is that the need to stay in the numpy sandbox really constrains how you choose data structures and algorithms.

Storing N-dimensional points in a Point class, for example, is often so slow as to be a non-starter. Admittedly, it's not just a Python problem: struggles with array-of-structs vs. struct-of-array representations are pretty ubiquitous.