75
I was wrong about robots.txt (evgeniipendragon.com)
submitted 1 year ago by to c/technology@lemmy.world
22 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] 79 points 1 year ago (2 children)

What did he think a crawler is? Why was he surprised that not allowing companies to use his data lead to them not using his data? Looks like he has another surprise coming when he notices that search engines no longer index his blog.

  • permalink
  • fedilink
  • source
  • hideshow 2 child comments
    •
  • [–] 16 points 1 year ago* (1 child)

    I feel like most casual users would not make the connection of "crawlers" to link previews that they talk about it the article.

    Sure, if you understand that robots.txt includes all robots then sure. But that is not how general news media has been talking about robots.txt.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •