51
Internet Archive to New York: Don’t Kill the Good Bots in the Fight Against Bad Bots | Internet Archive Blogs (blog.archive.org)
submitted 2 weeks ago by to c/linux@programming.dev
8 comments fedilink hide all child comments
 

cross-posted from: https://realbitcoin.cash/post/349131

I guess Anubis also prevents "good" bots such as search engine bots to access a webpage...

you are viewing a single comment's thread
view the rest of the comments
[–] 5 points 2 weeks ago

Anubis should add an option to "trust" certain tokens or public keys, so good scrapers like Internet Archive can get past.

  • permalink
  • fedilink
  • source
    •