this post was submitted on 29 Jan 2025
599 points (96.7% liked)
Technology
61227 readers
4350 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related content.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, to ask if your bot can be added please contact us.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Same problems with tarpitting. They search engines are doing the crawling for each of their own companies, you don't want to poison your own search results.
Conceptually, they'll stop being search crawls altogether and if you expect to get any traffic it'll come from AI crawls :/
I think to use it defensively, you should put the path into robots.txt, and only those doesn't follows the rule will be greeted with the maze. For proper search engine crawler, that's should be the standard behavior.
Spiders already detect link bombs, recursion bombs, they're capable of rendering the page out in memory to see what's truly visible.
It's a great idea but it's a really old trick and it's already been covered.