75
I was wrong about robots.txt (evgeniipendragon.com)
submitted 1 year ago by to c/technology@lemmy.world
22 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] 9 points 1 year ago* (last edited 1 year ago) (10 children)

See here: https://arstechnica.com/tech-policy/2025/07/cloudflare-wants-google-to-change-its-ai-search-crawling-google-likely-wont/ If you have a source that says it's false, I'd be curious.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 10 child comments
    •
  • [–] 2 points 1 year ago* (last edited 1 year ago) (9 children)

    Ok. That quotes a tweet by Cloudflare's CEO. IDK what his qualifications are, but his conflict of interest is obvious enough. Real quality journalism there.

    ETA: I looked at what the Cloudflare CEO said again. To be fair to him, he is not actually claiming that Googlebot collects AI training data. He's talking about the AI overview, which is a search feature. The data for search features is collected by Googlebot. I'm not sure why someone would want their link listed in search but not appear much more prominently in the AI overview.

    Here's Google technical documentation on its crawlers: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 9 child comments
    •
  • [–] 3 points 1 year ago* (8 children)

    So what's the quote from the documentation that backs up your claim? The line "perform other product specific crawls" seems extremely vague by design.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 8 child comments
    •
  • [–] 2 points 1 year ago (7 children)

    I'm not really sure what you are asking here. Did you notice that you can scroll down and see a list of their crawlers?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 7 child comments
    •
  • [–] 2 points 1 year ago* (6 children)

    Nothing on this page seems to contradict the article. But if I simply missed the part that does, I'd be happy to learn.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 6 child comments
    •
  • [–] 2 points 1 year ago (5 children)

    You look up what Googlebot does. No AI.

    You want to know what crawlers do AI? Just search for "AI", or "training", or some such, or skim through. It's not long. Google-Extended collects training data. Note that Google-Extended is explicitly not used to rank pages.

    Did that help?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 5 child comments
    •
  • [–] 2 points 1 year ago* (last edited 1 year ago) (4 children)

    You look up what Googlebot does. No AI.

    The page seems written to perhaps suggest it but doesn't explicitly say the other bots can't feed into some other sort of AI training. It would be in Google's interest to mislead the users here.

    Edit: I found a quote where it says Googlebot does both in one: "Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent [...]" and I guess Cloudflare doesn't trust Google to abide by the access controls. That seems sensible to me. Edit 2: What exactly the CEO believes was perhaps rightfully disputed below, it was just my guess.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 1 point 1 year ago (3 children)

    It would be a lot to write, if you had to say what something does not do rather than what it does.

    I looked at what the Cloudflare CEO said again. To be fair to him, he is not actually backing you up. He's saying that Google makes no difference between the AI overview and the other search results. That is true. The AI overview is a search feature. I'm not sure why someone would want their link listed in search but not appear much more prominently in the AI overview.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 1 point 1 year ago* (last edited 1 year ago) (2 children)

    But the article later does back it up: "Although Cloudflare singled out Google, other search engines that view AI search features as part of their search products also use the same bots for training as they do for search indexing."

    In any case, I'm okay with admitting neither you nor me can look inside Google to see they're doing. But the claims are out there, I didn't make them up, whether they're true or not. Thank you for the certainly interesting Google crawler info link.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 2 points 1 year ago (1 child)

    But the article later does back it up

    The CEO of Cloudflare did not assert that. I was surprised that he would claim such a thing, and that should have made me read more carefully. Elon Musk notwithstanding, neither incompetence nor conspiracy theorizing are common at that level, publicly anyway.

    You can believe whatever you like, of course. Freedom of opinion is nothing if not the right to be wrong.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •