40
How hard is it really to have your own webcrawlers and index? (lemmy.world)
submitted 18 hours ago* by to c/asklemmy@lemmy.world
23 comments fedilink hide all child comments
 

With the frustrations from using the usual search engines growing I've been thinking about this a lot lately.

It seems we've poisoned the well by allowing the proliferation of advertising interests to dominate the web.

Like how hard would it be to make your own non-commercial index?

Only human made sites that aren't related to buying, selling, marketing, etc.

Could that be a federated open-source project?

you are viewing a single comment's thread
view the rest of the comments
[–] 2 points 6 hours ago (1 child)

spin up YaCY, see for yourself. It turns into a full time project, because there's always ranking tweaks and and indexing startpoint changes to be made.

It's relatively easy to put together a crawler for specific purposes, but a general web search is hard, and requires a lot of storage.

  • permalink
  • fedilink
  • source
  • hideshow 1 child comment
    •