you are viewing a single comment's thread
view the rest of the comments
[–] 12 points 3 years ago* (last edited 3 years ago)

I tried finding information on what indexer they are using. Are they using their own?

Edit: says this in the readme:

The commoncrawl organization for crawling the web and making the dataset readily available. Even though we have our own crawler now, commoncrawl has been a huge help in the early stages of development.

  • source