I try to avoid using a graphical display while doing deep learning (or any gpu intensive work). It reserves some, albeit not much, vram and while it shouldn't cause issues like the one you're stating, it definitely can.
Sshing to the machine from another makes it far more consistent in that regard.