Nice project! Your 3D printed case for the stereo pair removes a lot of potentially useful information for the feature extraction algorithm. If it can help, I have an old project where we mounted a realsense (R300 if I remember well) on Hololens.

You can inspire from our 3D printed mount, the CAD is available here : http://vision.gel.ulaval.ca/~jflalonde/projects/hololens3d/i...