I don't know if I'm answering, but few years ago I've figured out and started to test this hypothesis:
Memory is context-driven, and such is our own ability to retrospect about spent time.
For example, let's say I spent whole Saturday doing one of my favorite combos; playing Factorio and listening to podcasts. Next day I would go to a dinner with a friend who (as most people on the planet) is not really interested in neither of these things. There's no way I could justify day spent, to my friend it would look like time wasted. Thing is, it's actually easy to come to a similar conclusion just myself -- I would feel like from some "objective", "classic" point of view, the time spent in Factorio was wasted.
However, one thing is easily missed: due to the contextual nature of our memory, the memory spent in one mindset (playing Factorio with podcast) is not readily available outside that mindset. (It has to be like this to some extent, right? we don't need to remember how to ride a bike when not close a bike!)
It sometimes happens to me that when I open old map from Factorio, memories from "the Factorio mindset" would start coming (including topics from podcasts or audiobooks), as if I visited some old place. If my friend walked up to me while I'm playing Factorio and asked me about how I spent my time, I could probably share lots of stories about how I came up with this structure and how I found myself stranded among enemy bases, etc. It's he change of context that prevents me to do so at the Sunday dinner -- part of the new context is that I'm with someone who's not interested in Factorio or podcasts.
The question is then, do all these experiences contribute in a positive way to something more long-term, like my personality? While playing/listening, am I training something that is going to be useful later on? It boils down to comparing what else could I have done, which is ultimately a futile enterprise anyway.
TL;DR: Could it be that in retrospect time can feel wasted but it's just because we're trying to "reach" the time from another context? Maybe we always spend our time the best way we can, it's just that we're not equipped to judge the time properly, at least not from any context.