Full title: I hid a link on my website that only bots can see and told them not to follow it. They broke the rules and in 4 days they went 124,414 levels deep into an endless maze. One bot opened 782,580 doors. I thought this was super interesting, this is how the internet is these days. Thought I’d share.

you are viewing a single comment's thread
view the rest of the comments
[–] 17 points 5 days ago (4 children)

Can anyone explain what is happening behind the scenes with the "room creation"?? That sounds intriguing. I understand OP is a bot but, is anything described here an actual way of trapping crawlers? How are the crawlers fooled by any of this?

  • source
  • hideshow 4 child comments
  • [–] 30 points 5 days ago (3 children)

    He placed an invisible link on his website. Only visible by reading the page content (html). Then the bots followed the link. This link lead to a page with 5 links (door) four links are dead and one leads to another page (level) with again 5 links.

    I guess the links and pages are generated on the go. So the bots and crawlers may follow the links forever of they like.

  • source
  • parent
  • hideshow 3 child comments
  • [–] 1 point 2 days ago (2 children)

    I wonder if pages have to be generated individually to appear legit to the crawlers, or if the same contents can be served as different links over and over... if I knew more about how this stuff works I'd try it myself hah.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 1 point 18 hours ago

    Most crawlers don't care - they're purely downloading. Most garbage will also be filtered out before training, and generating plausible garbage is expensive - not really worth the trouble. What you can do with the garbage, is that you can place unique identifiers in the links you serve, so when a crawler that you can't recognize otherwise comes visiting a poisoned URL, you can firewall it off. This works because the crawlers use a queue: they get a list of URLs to visit, they download them, and later on, the URLs from those get extracted, and crawlers will visit those URLs.

    So if you manage to trap a bot, and overwhelm its queue with poisoned URLs, you can also catch bots that you wouldn't be able to easily identify otherwise. And that is why serving an endless maze of garbage is so very useful. Because it lets you catch stuff like headless chromes operating through residential proxies: because those too, follow a queue.

  • source
  • parent