24
What's a good way to index the web? (feddit.org)
submitted 1 day ago* (last edited 1 day ago) by to c/opensource@lemmy.ml
12 comments fedilink hide all child comments
 

About a decade ago, I tried to index the web via the Dewey Decimal System. I had a site laid out similar to Google, where you could browse sites continuously starting from a given call, but the DDS is proprietary, and those people hate anyone who uses their IP without a license. You can Google everyone they've shutdown -- places that weren't even libraries -- for using anything similar to the DDS. I reached out to the group that manages the DDS, and was taken offline before my project even started.

With all the corporate BS lately, and people looking for alternative options, I thought I'd take a search engine old school, and we'd index like Usenet. Except with an XXX.XXX.XXX format.

Problem is, now I've got to find categories, sub categories, and local categories. Categories for animals, chat rooms, cars, construction, wrestling, etc.

I need a brilliantly created categorical system with a TLD.MLD.LLD (top level subject domain, mid level domain, and local level domain) format.

I was going to let site owners Disney Land it, by letting them create an index.json file in the root of their site, choosing for themselves where they wanted to appear in the index. Problem is, as trends change and systems become more exact, site owners would have to update their index. Which could become cumbersome.

Further, if we divided into academic disciplines, there would be things like gaming and model construction which wouldn't fit into those.

It would help if there were an already established categorization system, so that site owners didn't have choice anxiety. And then, as the site grew, site owners could recategorize themselves with user feedback.

Could you guys suggest a few categorization systems (can't be DDS), either of your own making or other sources?

I'm already looking at Usenet: https://www.livinginternet.com/u/uw_hier.htm

Edit: Current UX: https://imgur.com/a/sURm4s1

  • Haven't chosen a name yet.
  • Adult is probably going to be dropped, and when the site owner chooses a category, they can put it in the ADT TLD.

all 12 comments

sorted by: hot top controversial new old
[–] 2 points 9 hours ago

Bless your heart OP, but attempting to hierarchically taxonomise all of the internet is just futile.

And not just even talking about the infrastructure required, but the fact that pinning ideas into a taxonomy or ontology frames them in image of the taxonomising taxonomer, and not in the one querying it.

It is also one of the reasons that the ideas of Web 2.0 never truly caught on, and why the population of json users dwarfs that of RDF users.

Perhaps instead of a hierarchical categorisation you could use vector embeddings with a similarity measure. That frees you from the burden of taxonomising and perhaps leaves you more energy to refine your indexed corpus for relevance and utility.

Best of luck!

  • permalink
  • fedilink
  • source
    •
    [–] 2 points 18 hours ago*

    You can't use the DDC because classification systems require a custodian who is in charge of maintaining the system as the dataset changes.

    For the same reason you can't just slap any UPC code on an item and sell it. There are standards for a reason.

    You would need to invent your own classification system, and a mechanism that would not only be able to automate the process but also process the dataset quickly enough to keep up with change rate of the internet.

    The infrastructure for a project this size would demand a buisness model to finance its upkeep. Its impractical in the same way trying to run your own search engine index isn't really practical, which is why metasearch engines like SearX are a thing.

    Maybe start with something more realistic and achievable like indexing the Lemmy Fediverse.

  • permalink
  • fedilink
  • source
    •
    [–] 6 points 1 day ago

    I don't understand. The DDS is old enough to be out of copyright and patents aren't nearly as long-lived, so what are these people using to restrict your right to it?

  • permalink
  • fedilink
  • source
    •
    [–] 9 points 1 day ago (6 children)

    This is an insanely complex problem to solve. Companies like Google have spent billions on figuring this shit out. If you are good at these kinds of things and do something novel, you have a six figure job waiting for you at one of these companies.

  • permalink
  • fedilink
  • source
  • hideshow 6 child comments
    •
  • [–] [S] 5 points 1 day ago* (4 children)

    Everybody's gotta start somewhere. Google started in a garage.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 7 points 1 day ago (3 children)

    that was apple.. and nirvana. google was a phd project

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
    • [–] 4 points 1 day ago (1 child)

    Cool idea! Although in the age of slop, I think your project would be more like indexing a curation of the web.

    Have you considered something like the Library of Congress system? https://en.wikipedia.org/wiki/Library_of_Congress_Classification Because of the inclusion of the Date, maybe maybe this could be used in conjunction with archive.org

  • permalink
  • fedilink
  • source
  • hideshow 1 child comment
    •
  • [–] [S] 2 points 1 day ago*

    I originally liked Dewey because it slid from one topic to the next. And, as you were transitioning, the fields just merged into one another. It wasn't obvious that 2 particular subjects were separate. My original plan was to use this to allow the user to go from page to page to page, after they'd originally chosen a starting position, they'd scroll pages like library shelves. I haven't found another system like that.

    The Library of Congress system isn't as intuitive as Dewey either :'(.

    I won't be using web crawlers. Probably will stick with the site admins choosing where they're indexed via an index.json file (or similar name).

  • permalink
  • fedilink
  • source
  • parent