you are viewing a single comment's thread
view the rest of the comments
[–] 41 points 1 year ago (1 child)

So. If I can add something here for everyone's benefit

No search engine really obeys robots.txt

Their publicly acknowledged crawlers do, but they have other crawlers that aren't know that ignore the file.

Google knows every inch of your site, allowed or not.

See, just because a search engine says it doesn't know, doesn't mean it hasn't crawled. Just doesn't display the results based on your settings.

  • source
  • hideshow 1 child comment