Instead of remembering what line number you were at, you can use marks (:help mark-motions) to immediately jump back to where you left off.
For example, type mx to mark the current position with x (or anything you want). Say now you are at the top of the file, just type 'x to go back to the line marked with x.