Any good web crawler has limits.
Yeah. Like, literally just:
- Keep track of which URLs you've been to
- Avoid going back to the same URL
- Set a soft limit, once you've hit it, start comparing the contents of the page with the previous one (to avoid things like dynamic URLs taking you to the same content)
- Set a hard limit, once you hit it, leave the domain altogether
What kind of lazy-ass crawler doesn't even do that?