All good questions I don't have good answers to.
Especially when you compare entirely different software (Lemmy vs Mastodon) that happen to both use ActivityPub. So there is already an imbalance. And the imbalance has also started within Lemmy where larger instances have a sort of snowball effect.
I think ideally the "too large" instances would find more admins and split the instance into smaller ones. But that might not be feasible today with the existing technology and human resources.
Another option is to "stop the bleeding" by locking an instance to new accounts to give others a chance to grow. This should have already happened on lemmy.ml IMO. The instance performs very poorly (as of a week ago).
how would you catch / block a no user relay node that relays data to a quarantined node?
I don't know the technical details of ActivityPub well enough to propose a real solution. Sounds like an arms race situation.