Memes

There are two ways of looking at it.

1. The time machine is using itself as a point of reference to comply with general relativity. The only way to time travel is to move forward in time. The way to move through time would be to move a lot faster than the Earth, so that every minute for you inside the time machine would equal to many years for earthlings. And if you're moving that fast you'll fly away from Earth.
2. The time machine somehow has a knowledge of the whole universe, this way a Newtonian model applies and an absolute point of reference exists. That allows unrestricted travel both forwards and backwards in time, but that also means that the Earth will inevitably move from under the machine to follow its path across the universe.

No matter how you twist it you'll end up all alone in space. You need a machine which can move through both time and space at the same time.