There's a ton of issues with time travel. That could be one, but most fictional time-travel devices can be said to accommodate for the difference in distance. It would just be boring to explain on-screen.