40
How hard is it really to have your own webcrawlers and index? (lemmy.world)
submitted 16 hours ago* by to c/asklemmy@lemmy.world
23 comments fedilink hide all child comments
 

With the frustrations from using the usual search engines growing I've been thinking about this a lot lately.

It seems we've poisoned the well by allowing the proliferation of advertising interests to dominate the web.

Like how hard would it be to make your own non-commercial index?

Only human made sites that aren't related to buying, selling, marketing, etc.

Could that be a federated open-source project?

you are viewing a single comment's thread
view the rest of the comments
[–] 29 points 15 hours ago (1 child)

Surprisingly easy. This guy did runs his own search engine that focuses on the "marginalia" of the internet, i.e. smaller sites:

https://marginalia-search.com/

Business model

Web search has traditionally been difficult to monetize, which has pushed many search engines to go the route of advertisement, to the detriment of the search results.

The project is independent in that it has no loans, no investors looking for a payday, no strings attached anywhere to pressure it into doing anything than providing as much and as good internet search as it is capable of.

The marginalia search engine is designed to be very cheap to run and operate, and the goal is to provide outsized value, and thus be able to scrape by on donations, grants and commercial API-deals with other search engines.

The project currently has bills in the ballpark of $200/month, meaning it can keep operating even if funding runs completely dry, although this would cause development to stall almost completely.

  • permalink
  • fedilink
  • source
  • hideshow 1 child comment
    •