The crawler did not respect our robots.txt
But was still allowed to proceed. That's like... not using the streets in a public park.
The crawler did not respect our robots.txt
But was still allowed to proceed. That's like... not using the streets in a public park.