It's a valid issue. You wouldn't need to worry about any component of your motion that's "constant velocity" (so, inertial straight-line motion), since all inertial reference frames are equally valid. But you would need to keep track of curving motion (like the rotation of the Earth, orbit around the Sun, Sun circling around the galactic core, etc.).

You might be able to wave your hands and claim that the time machine follows paths back in time along natural world-lines in curved space-time (gravitational geodesics), which will at least get things like orbiting around the Sun right (and galactic motion as well). But I think you'd still need some way to keep the machine's path rotating on the surface of the Earth rather than falling back and forth through the center of the planet.