51
Internet Archive to New York: Don’t Kill the Good Bots in the Fight Against Bad Bots | Internet Archive Blogs (blog.archive.org)
submitted 2 weeks ago by to c/linux@programming.dev
8 comments fedilink hide all child comments
 

cross-posted from: https://realbitcoin.cash/post/349131

I guess Anubis also prevents "good" bots such as search engine bots to access a webpage...

you are viewing a single comment's thread
view the rest of the comments
[–] 3 points 2 weeks ago

--user-agent "Mozilla/5.0 (X11; Linux x86_64; rv:154.0) Gecko/20100101 Firefox/154.0 Scrapping-For-Research-Hope-Its-Okay"

Here, It's disclosed, no?

  • permalink
  • fedilink
  • source
    •