On a whim, a moderator could decide to defederate with one of your favorite servers for no reason at all.
Sure, that could happen. Mostly though, it's because your favourite server is some variety of bigot bar and you really want to hang out with bigots.
Every single time people want to have an unmoderated, unchecked space, it's because they — you — want to be able to air your most unsavoury opinions with impunity.
Then come the Nazis, then come the CSAM, the decapitation videos, the harassment and doxxing, and so on until some part of it passes your personal line in the sand for what's acceptable.
Then all of a sudden it's "Wah wah, mommy give me moderation". It's the same for all this freeze peach crap.