Obviously I'm wrong but I'm not sure why. Light bouncing off of two mirrors can interfere constructively or destructively, depending on the phase difference. The phase difference from light bouncing off of two aligned mirrors is random (as it's non coherent light). Adding a constant in the phase difference due to misalignment would still lead to a random difference. So it seems to me the phase shift wouldn't make a difference.
What am I missing?