Maybe it depends on where you live, but I always find the same spiders living in gardens or on the side of a shed or something, so I've never been worried about that except in winter.

Does not apply so much in warm weather countries. All the spiders that come into our house are also easily found loving life outside (Australia). I just gently move them out, except daddy long-legs, generally let some of them stay as pest control.

wolf spiders not so much