EDIT: changed to more accurately represent how Matrix operates.
The issue is that due to the way Matrix is structured, it essentially spreads copies of unencrypted metadata to every instance participating in those rooms, So it's federated, but difficult to actually keep metadata from being spread around even if you don't federate with the main Matrix server, if any server you do federate with dies, it'll get spread there. You'd have to be extremely cautious who you federate with to avoid that, or not federate at all, which defeats the purpose.
As an alternative, Movim, which uses XMPP and is also federated, does not spread meta data around like that.