Full title: I hid a link on my website that only bots can see and told them not to follow it. They broke the rules and in 4 days they went 124,414 levels deep into an endless maze. One bot opened 782,580 doors. I thought this was super interesting, this is how the internet is these days. Thought I’d share.
Most crawlers don't care - they're purely downloading. Most garbage will also be filtered out before training, and generating plausible garbage is expensive - not really worth the trouble. What you can do with the garbage, is that you can place unique identifiers in the links you serve, so when a crawler that you can't recognize otherwise comes visiting a poisoned URL, you can firewall it off. This works because the crawlers use a queue: they get a list of URLs to visit, they download them, and later on, the URLs from those get extracted, and crawlers will visit those URLs.
So if you manage to trap a bot, and overwhelm its queue with poisoned URLs, you can also catch bots that you wouldn't be able to easily identify otherwise. And that is why serving an endless maze of garbage is so very useful. Because it lets you catch stuff like headless chromes operating through residential proxies: because those too, follow a queue.