The way I solve this problem is to use nnn instead of ls. It's a full file manager for the terminal but so light and quickly started that it's as convenient as ls (a lot more so actually). You can navigate around your file system with vim-like key bindings and then have the option to exit either to the directory you started from or the one currently displayed.