The ones that are difficult to block use fake useragents, however there are WAFs that keep track of bots that do aggressive crawling and block those. Usually, that, combined with rate limiting rules, blocks most of them.

You are correct though, it cannot be completely prevented unless you restrict it to users and disallow user creation (like private torrent trackers do).