188

Now even my personal instance is infested by these AI crawlers (potato-guy.space)

submitted 1 day ago* (last edited 1 day ago) by [email protected] to c/[email protected]

29 comments fedilink hide all child comments

It's impossible, i got this instance to just see lemmy from my own instance, but no, it was slow as hell the whole week, i got new pods, put postgres on a different pod, pictrs on another, etc.

But it was slow as hell. I didn't know what it was until a few hours before now. 500 GETs in a MINUTE by ClaudeBot and GPTBot, wth is this? why? I blocked the user agents, etc, using a blocking extension on NGINX and now it works.

WHY? So google can say that you should eat glass?

Life is now hell, if before at least someone could upload a website, now even that is painfull.

Sorry for the rant.

you are viewing a single comment's thread
view the rest of the comments

[-] [email protected] 4 points 12 hours ago* (last edited 8 hours ago)

Just cache. Read only traffic should add negligible load to your server. Or you're doing something horribly wrong

[-] [email protected] 5 points 12 hours ago

They are 1 cpu and 1 gb of ram pods, postgres goes to 100% cpu on 500 requests per minute, after i put the NGINX extension, it reduced to at max 10%. On weaker servers, these bots make hell on earth, not the config.

[-] [email protected] 5 points 11 hours ago

If it's hitting postgres it's not hitting the cache. Do you have a caching reverse proxy in front of your web application?

[-] [email protected] 1 points 10 hours ago* (last edited 10 hours ago)

I don't have a cache, but the problem is solved now, i can browse lemmy haha.

[-] [email protected] 4 points 10 hours ago

The nginx instance you have in front of your app can perform caching and avoid hitting your app. The advantage is that it will improve performance even against the most stealthy of bots, including those that don't even exist yet. The disadvantage is that the AI scum get what they want.

[-] [email protected] 2 points 10 hours ago

Oh, cool. I'm going to look at it!

[-] [email protected] 2 points 8 hours ago

If that doesn't work for you, also look at varnish and squid.

load more comments (2 replies)

load more comments (3 replies)

this post was submitted on 08 Jun 2025

188 points (98.0% liked)

Fuck AI

3034 readers

1015 users here now

"We did it, Patrick! We made a technological breakthrough!"

A place for all those who loathe AI to discuss things, post articles, and ridicule the AI hype. Proud supporter of working people. And proud booer of SXSW 2024.

founded 1 year ago

MODERATORS

[email protected]