Correct, but that doesn't change who has final say over it. You're more than free to change instances if you no longer agree with how your current instance is being run.

I'm pretty sure this has already happened in the past, I swear there was a "Bad instances" list that someone was passing around, which someone else then disputed that the list was bad due to there not being valid reasons for why an instance was marked "bad" - but people had taken the list as 100% fact and blocked said instances.

That is the double-edged sword of the Fediverse, the freedom to choose who you allow and don't allow in regards to federation. There's always going to be the "cliques" so to speak where if you upset (for lack of a better word) the wrong people, the size of that instance can claim that you're bad, and if other instances take their word at face value without verifying this then all of a sudden you can't communicate with other instances/people (ie, if you get defederated by lemmy.world or mastodon.social - then good luck). Obviously, the good part about how the Fediverse works is the power for each instance admin to make their own determinations of who they want to federate with, but this is the "bad" side of it which is further amplified by the fact that there are always going to be instances that hold a very larger position of power. In a way, fracturing of the Fediverse is a bit inevitable because of this. I suspect what you say will happen (as I've already seen this mentioned).

It is what it is, I'm not saying whether that double-edged nature is good or bad, because at the end of the day that determination comes down to every person who chooses to participate, and is a decision they have to make on their own volition.