601
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 13 May 2025
601 points (98.9% liked)
Technology
71355 readers
3551 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 2 years ago
MODERATORS
Lmao you linked to the same page I did where this text appears:
Also you're so capitalism brained you assume anyone running a website must be doing so for profit. My hobby projects (personal homepage and personal git forge) were getting slammed by bots while I just paid the bills. I could have locked them both behind an auth portal but then I might as well just take them off the internet and run everything on my LAN.
So far none of your ramblings disproves what I said. Yeah there are crawlers for niche collecting probably, nobody crawls the entire internet when they can use the weekly updated common crawl. Unless you or anyone else has access to unknown internal openAI policies on why they intentionally reinvent the wheel, your fake anecdotes (lol bots literally telling you they're going to use scraping for training in the user agent) don't cut it. You're probably seeing search bots.
If you didn't care for ad money and search engine exposure bozo you'd block everything in robots.txt and be done instead of whining about specific bots you don't like.
You didn't link to this but go on take their IPs json files and block them.