this post was submitted on 27 Aug 2023
79 points (100.0% liked)

Free and Open Source Software

17970 readers
53 users here now

If it's free and open source and it's also software, it can be discussed here. Subcommunity of Technology.


This community's icon was made by Aaron Schneider, under the CC-BY-NC-SA 4.0 license.

founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
[–] [email protected] 12 points 1 year ago* (last edited 1 year ago)

I tried finding information on what indexer they are using. Are they using their own?

Edit: says this in the readme:

The commoncrawl organization for crawling the web and making the dataset readily available. Even though we have our own crawler now, commoncrawl has been a huge help in the early stages of development.