I'll give a half-baked counter to this: we know gravity impacts the flow of time through relativity. There is currently no evidence that time travel wouldn't be impacted by gravity in some way. Maybe the way time in time travel interacts with gravity protects you from this problem? Probably not, but it has just as much evidence to support it as your claim of time travel will dump you in empty space.