It's not really the previously banned users that are the problem. It's that the real heart and soul of Lemmy is c/2real4meirl or whatever - ie, depression memes.
Reddit initially became popular because it was fun and interesting. Lemmy has picked up some of the old reddit crowd by being a bit more tech focused - but for the most point the links and comments posted are doom and gloom. Either AI is taking all our jobs, or its a huge scam. The world is run by evil capitalists who personally want you, in particular, to have a meaningless and miserable life. But don't worry, because we, the proletariat, will overthrow them in a violent revolution... just as soon as we stop doom scrolling and crying in bed - haha, amiright guys?
Nothing about this is fun or interesting. It is bitter, angering, and depressing. That is what drives people away.