That's how I feel when I'm not invested in my free time. When I have a video game that I'm trying to finish, or an activity planned after work, I jump into it and spend my time pretty well. (I don't care what you think, I'm considering 5 hours of progress in Final Fantasy a good use of time.)
When there's no plans, and nothing in particular that's grabbed my interest, I kind of just sit and wait for something to happen.