Yeah, they're not sensitive to magnetic fields. But they do drift with temperature. Which leads to fancy semi-improv solutions, like placing the quartz in a small case, together with a heating element and a temperature sensor, to keep that sucker incubated at constant temp. A lot of HAMs used to do this back in the day - or maybe they still do, I dunno.
Also, placing a small variable capacitor in series with the quartz is an easy way to tune it, if it's too fast. I actually have a wall clock that looks nifty but is too fast, and I'm gonna tune it by trial and error, inserting small caps in series with the crystal.