Number of (active) Lemmy users seems to stabilize and I think this is a great thing. Indeed we got a lot of users when reddit shutdown its API (I was among them despite being a long time oss user), many have left, but the community seems now to stabilize to ~ ½ of the big grow in june '23. I think this is very nice for lemmy, we can be proud of this project.
The stats come from: https://fedidb.org/software/lemmy
For me it's more about something with the potential to undermine the deathgrip of facebook, twitter, etc from our societies, in an attempt to partially address the steady increase in utter braindead stupidity and mindless vitriolic hatred stemming from profit-driven algorithmic control of our broader information ecosystem.
That requires the eventual growth of a decentralized sister ecosystem of sorts, able to act as a viable competitor. So, to compete with the giants, that's basically a fairly significant chunk of everyone. On Earth.
Fortunately you can always just defederate the highest population centers, I imagine that would become very common eventually. Especially if they tried monetization or something, which would probably not be unheard of with larger userbases.
All that said, I do feel you and also personally like smaller communities. But I'll just move to one once World gets too big.