Your knock on the door analogy is exactly right--when I started my instance, I had to search every community that I wanted to see directly by URL. Then my server would send a message to that community's server saying that I subscribed to that community. Now, every time a post is made at that community, it's server sends my server an update. If I post a comment to a community on lemmy.ca (like I am now), from my kbin instance (remy.city), and you are reading it from kbin.social, that means my server first saved my comment locally, then sent it to lemmy.ca, and lemmy.ca sent it to your kbin.social because you subscribed to the community. So in that case, lemmy.ca is the 'authority', and is responsible for sending updates out to subscribed parties.

There is no such thing for instances--each new instance has to manually make a connection to another (i.e. a user on the new instance must subscribe to something from another instance). I think the tools like fediverse.observer are reading comments or other activity from popular instances, and are then compiling a list of the instances they find by doing that. But there is no central server/authority that makes communication between instances possible. Each instance has to talk to each other instance for it to happen. It's a bit inefficient but is necessary for decentralized communication.

I mean, there has to be some sort of system somewhere that helps the servers connect to each other. How does Lemmy.ca know that Lemmy.world exists? There must be some sort of authority that knows. There must be some sort of first step when a new instance appears that lets everyone know that the new server exists.

There literally isn't. New servers do not automatically federate with each other. Someone on the new server needs to manually start following users or groups on existing servers just to to establish any kind of connection. And even then, people on the existing server won't know that any users or groups exist on the new one.

It takes conscious effort by users to create connections and start content flowing between fediverse websites. There's no central authority of any kind. If someone doesn't make those connections, a fediverse website is functionally a stand-alone social media website.

Yes, it's literally just like that. You have to announce to the fediverse you're open to federate with them and then they have the ability to defederate whenever they want.

The way I've seen this work previously on fedi is that people post "hey I made a new server, please boost for reach". That effectively announces the existence of the server to the network. It can be difficult to get noticed at first, if you are a single-user instance without many followers.

Think of it like email vs a website. Microsoft.com isn't connected to Yahoo.com, but they store email originating from both places. The difference here is those emails are email lists (posts and comments from subscribed communities) and get sent automatically once they learn each other exist (because a user asked to get an email).