Interesting find about the indexing. I just had the opposite experience, swapped from numpy to torch in a project and got 2000x speedup on some indexing and basic maths wrapped in autodiff. And I haven't moved it onto cuda yet.
import time, torch
from itertools import product
N = 100
ten = torch.randn(N,N,N)
arr = ten.numpy()
def indexTimer(val):
start = time.time()
for i,j,k in product(range(N), range(N), range(N)):
x = val[i, j, k]
end = time.time()
print('{:.2f}'.format(end-start))
indexTimer(ten)
indexTimer(arr)