My favorite patch (other than "what frame of reference are you using for this and why that one in particular, none of them are more correct than any other") is to just say gravity continues to affect you even as you're traveling in time, effectively holding you in the same place if you're on a planet or in orbit.