That's how I always think about it too. Imagine telling someone "I jogged all the way to the store". Assuming it's more than a mile away, that might be a little surprising. If the person then says, "well I biked, but same point, I got there", it's a slightly different reaction. "I jogged" and "I biked" are two different things with different meanings.

Not that it should be looked down on or it's bad at all.

I guess there's a gap in the language a bit. There's not a great sounding word (afaik) to indicate 'consuming a book in whatever format', so it's understandable that 'read' evolves into that. Still, in most cases to me it makes more sense to choose between 'read' and 'listened'.