139

submitted 3 months ago* (last edited 3 months ago) by throws_lemy@lemmy.nz to c/tech@programming.dev

9 comments fedilink hide all child comments

you are viewing a single comment's thread
view the rest of the comments

[-] pivot_root@lemmy.world 8 points 3 months ago

"robots.txt files? You mean those things we use as part of the site index when scraping it?"

— AI companies, probably

this post was submitted on 11 Nov 2025

139 points (98.6% liked)

Tech

3135 readers

2 users here now

A community for high quality news and discussion around technological advancements and changes

Things that fit:

Things that don't fit

founded 2 years ago

MODERATORS