May be worth keeping some local communities in that case, which can also serve as sort of backup for wider community from other small servers. For example, if there is "Knitting" on a big instance, you can consider creating something more specific like "Knitting full RGB sweaters" on your smaller instance. Then there is a basis for sustained discussion there, and more people can come if something breaks. I have some ideas for comms like this that I'll maybe come around to creating.
I don't think we need to keep full centralization be-where-everyone-is mentality here. Or maybe be where everyone is, but don't make it the only place where you talk with people.