I don't know rust but rust seems very much like C++.
Did you have a look at the Eigen library (C++)?
http://eigen.tuxfamily.org/dox/TutorialMatrixClass.html
I have never worked with this library but it seems to me that they have not the problems you described. Maybe having a look at it brings up some new ideas...