Someone on lemmy suggested to create a dummy endpoint that normal people won't be able to navigate to, and disallow it in robots.txt
Then when somebody crawls it you know they are ignoring robots.txt, and you ip ban them
Someone on lemmy suggested to create a dummy endpoint that normal people won't be able to navigate to, and disallow it in robots.txt
Then when somebody crawls it you know they are ignoring robots.txt, and you ip ban them