Selfhosted

62,003 readers

5,500 active

1,156 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

Detailed Rules Post

Be civil.
No spam.
Posts are to be related to self-hosting.
Don't duplicate the full text of your blog or readme if you're providing a link.
Submission headline should match the article title.
No trolling.
Promotion posts require active participation, with an account that is at least 30 days old. F/LOSS without a paywall has exceptions, with requirements. See the rules link for details. Tags [CBH] or [AIP] are required, see the links in Rule 8 for details.
AI-related discussions and AI-involved promotional posts have additional requirements for tagging, as noted in Rule 7 and the AI & Promotional Post Expanded Rules post, and find example disclosures here.

Resources:

selfh.st Newsletter and index of selfhosted software and apps
awesome-selfhosted software
awesome-sysadmin resources
Self-Hosted Podcast from Jupiter Broadcasting

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 3 years ago

moderators

curbstickle@anarchist.nexus

curbstickle_lw@lemmy.world

Based on this graph, and this graph alone, guess at what time I completely blocked OpenAI crawlers (thelemmy.club)

submitted 6 months ago by hylobates@jlai.lu to c/selfhosted@lemmy.world

79 comments fedilink hide all child comments

I really hope they die soon, this is unbearable…

A graph of MySQL queries showing a violent drop in requests after blocking OpenAI Crawlers

you are viewing a single comment's thread
view the rest of the comments

[–] AHemlocksLie@lemmy.zip 17 points 6 months ago (3 children)

Pretty sure I've repeatedly heard about the crawlers completely ignoring robots.txt, so does Cloudflare really do that much?

permalink

fedilink

source

parent

hideshow 3 child comments

[–] Sv443@sh.itjust.works 9 points 6 months ago (1 child)

Like a lock on a door, it stops the vast majority but can't do shit about the actual professional bad guys

permalink

fedilink

source

parent

hideshow 1 child comment

[–] FreedomAdvocate@lemmy.net.au 2 points 6 months ago

Cloudflare definitely can and does stop the vast majority of actual professional bad guys.

permalink

fedilink

source

parent

[–] tomjuggler@lemmy.world 6 points 6 months ago

Yes, CloudFlare blocks agents completely if they ignore it's restrictions. The key is scale - CloudFlare has a birds eye view of traffic patterns across millions of sites and can do statistical analysis to determine who is a bot.

I hate the necessity but it works

permalink

fedilink

source

parent