Agh, you're quite right. Thanks for correction. I crossed my wires and misremembered how it worked after reading this article about it a while back. Edited my previous comment to reflect that.
I suppose in theory that shouldn't be an insurmountable problem, though in Matrix's case it's a big roadblock, as the main Matrix server hosted by Matrix themselves has unfortunately become the defacto main server that most people use, which means not federating with it massively reduces the ability for someone to just be able to seamlessly hop onto your server unless they too are on one of the smaller, less popular servers.
In the example given in the video, it would likely be a bit of a deal breaker if you met someone in an online game somewhere, and then invited them to your self-hosted Matrix server, only to discover they are on the main potentially israeli intelligence-tied Matrix instance, meaning you'd have to explain they need to create an account elsewhere to be able to join your instance. It would be pretty awful UX.