We assume all of that.

We really don't know how common objects are between stars. We've observed a couple of interstellar visitors passing through our system, old asteroids from origins unknown. But we can only track the really large ones. We can see mid sized objects when they come very close to earth, but we can't track small objects at all, we literally have no idea how much of that exists in deep space. (Or for the most part, in our own solar system)

Still, you can see just from the evidence that our deep space probes have never really (hit) anything, that it would in fact be a quite rare event. It's just hard to say how rare.