I guess I am fine with slow steady growth rather than everyone wanting to jump over.
One thing I think is that you need to use Lemmy in a certain way. I remember reddit not clicking for me until Digg forced my hand, and that event likely irreparably changed the reddit crowd.
We probably have some way to go before we can solve the problem but somehow we need to teach people to aggressively block (people, communities, instances) that they don't want to see.
Or something I think is important, the ability and willingness to just stop replying when the conversation is no longer constructive.