40
How hard is it really to have your own webcrawlers and index? (lemmy.world)
submitted 16 hours ago* by to c/asklemmy@lemmy.world
23 comments fedilink hide all child comments
 

With the frustrations from using the usual search engines growing I've been thinking about this a lot lately.

It seems we've poisoned the well by allowing the proliferation of advertising interests to dominate the web.

Like how hard would it be to make your own non-commercial index?

Only human made sites that aren't related to buying, selling, marketing, etc.

Could that be a federated open-source project?

you are viewing a single comment's thread
view the rest of the comments
[–] 3 points 12 hours ago

Storing the index on the blockchain itself isn't really feasible, but I think you could leverage it as a source of truth for what distributed stores (e.g. torrent, WebDAV, ftp, s3) contain the correct and latest info.

Tough part is verification. You need a consensus for what is accurate before you add it to the chain. If someone indexes 1000 sites and someone else goes to verify it the sites could've changed in that time. Doesn't mean the proposed data is wrong but how do you separate outdated info from bad info (i.e. data poisoning)?

  • permalink
  • fedilink
  • source
  • parent