Web crawling has always worked through raw HTTP/HTML parsing, why create site specific API calls that require authentication and are throttled.

This excuse is pure bullshit.