Ugh. I've always liked Matrix (and was not bothered too much by the metadata leaks because my home server was not federated anyways), but after noticing some issues and finally reading up on the actual protocol spec a couple of weeks ago... oof. Yeah. No.
Set up XMPP for now. Works really well and the protocol seems so much saner. Unfortunately, it too has some annoyances that are unacceptable to me in the long term. I'm this close to saying "fuck it" and wasting the next couple of years of my life on a new protocol that no one is gonna use. (Cue the XKCD here.)