If I remember hearing it correctly, some lemmy instance admins defederate by proxy. Meaning: they go look at your instance, see who you federate with (even if you do it passively by not blacklisting) and then defederate you if you federate with any instances they don't like.
(And tbh, I'd do the same, but it doesn't seem like OP would like that)