In terms of impedance mismatch, I wish the PyTorch API was more similar to numpy. eg: using .shape instead of .size, using the same method names where possible. Seems like a small detail, but it could make PyTorch a little bit more intuitive.
> x = torch.rand(4, 3)
> x.size()
torch.Size([4, 3])
> x.shape
torch.Size([4, 3])
Their explicitly stated goal is to match numpy semantics as much as possible :-)