I'd argue it's legitimate for server owners to decide who their servers communicate with arbitrarily. Servers that want to be popular, however usually have consistent policies for moderation.

I self-host Mastodon and subscribed to a couple relays early on so my server has seen content from a variety of places. After I added a full-text search patch, I searched for some slurs to see if any racist content had reached my server. Some had, and it was all from poa.st - hardcore Nazi and KKK sort of stuff. I would absolutely block them if I was operating a public server.

Fortunately, those people don't try to interact with the bird photography and flashlight reviews my accounts post, so I haven't had to do any moderation.