Ok so you're assuming the position stays constant while time travelling. As mentioned, there is no absolute xyz, so whoever made the time machine had to choose a frame of reference in which the position would stay constant. The logical thing to do would be to take the frame of reference of the earth, and so time travel wouldn't cause any issues.