585
Based on this graph, and this graph alone, guess at what time I completely blocked OpenAI crawlers (thelemmy.club)
submitted 6 months ago by to c/selfhosted@lemmy.world
79 comments fedilink hide all child comments
 

I really hope they die soon, this is unbearable…

A graph of MySQL queries showing a violent drop in requests after blocking OpenAI Crawlers
you are viewing a single comment's thread
view the rest of the comments
[–] 27 points 6 months ago (8 children)

It's best to use either Cloudflare (best IMO) or Anubis.

  1. If you don't want any AI bots, then you can setup Anubis (open source; requires JavaScript to be enabled by the end user): https://github.com/TecharoHQ/anubis

  2. Cloudflare automatically setups robots.txt file to block "AI crawlers" (but you can setup to allow "AI search" for better SEO). Eg: https://blog.cloudflare.com/control-content-use-for-ai-training/#putting-up-a-guardrail-with-cloudflares-managed-robots-txt

Cloudflare also has an option of "AI labyrinth" to serve maze of fake data to AI bots who don't respect robots.txt file.

  • permalink
  • fedilink
  • source
  • hideshow 8 child comments
    •
  • [–] 18 points 6 months ago (3 children)

    If you're relying on Cloudflare are you even self-hosting?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 17 points 6 months ago (3 children)

    Pretty sure I've repeatedly heard about the crawlers completely ignoring robots.txt, so does Cloudflare really do that much?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 6 points 6 months ago

    Yes, CloudFlare blocks agents completely if they ignore it's restrictions. The key is scale - CloudFlare has a birds eye view of traffic patterns across millions of sites and can do statistical analysis to determine who is a bot.

    I hate the necessity but it works

  • permalink
  • fedilink
  • source
  • parent