522
Google Search is now using AI to replace headlines (www.theverge.com)
submitted 5 months ago by to c/technology@lemmy.world
117 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] 2 points 5 months ago (1 child)

Yeah, that's a problem. So even if you want to search your local newspaper, you won't be able to because of bot protection. Your crawler will be blocked. You also can't put the whole of Reddit or even Lemmy in your search index.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 5 months ago* (last edited 5 months ago)

    I do think this is one of the places where yacy has it's strength since yacy is p2p you can retrieve results from other nodes that have indexed the page thus large websites can be collectively indexed by many nodes.

    I also don't remember having problems with the crawler getting blocked but I also didn't do a lot of crawls I can see this being a problem though.

  • permalink
  • fedilink
  • source
  • parent