It was slow because at first I put the import inside the function. Moving it out improved the python performance to 20ms (as I mentioned in another comment).
The javascript version runs in 15ms.
> In the real world, things like haversine are implemented as C extensions
Having to write C-extensions is, well a point of friction, and adds complexity.
If you just used a native language to begin with, the simple straight forward code would work just fine.
Btw the source is here, and it also includes the csv file. https://bitbucket.org/hasenj/geodb/src