Often its easier to just scrape everything rather than making a whole code to pull api requests and put them into a database and sort them while doing so. These companies scrape the entire internet, they don't have time or necessarie to use api, they don't need permanent access to a two way communication, they just need the data.