I found it to be slow at times, but more annoyingly,
Slow at what, exactly? If you mean slow at delivering messages, it suggests that you were using the world's largest public server, which sometimes gets overloaded enough to be slow. In that case, your criticism is not of Matrix, but of a particular server. To compare apples to apples, you would have to either pick a different server or compare the largest one with a similarly loaded XMPP server.
it would very consistently not un-encrypt messages both for me and the people I was talking to,
When was that? Which clients were in use? This is relevant because unable-to-decrypt errors were fairly common until roughly mid-to-late last year. They put a lot of work into finding and addressing the causes, and I haven't seen a single one in more than a few months. I suspect the experience you're describing here is either out of date, or you're using clients that haven't applied the fixes yet.
I also notice from your recent Lemmy posts that you are evangelizing Movim pretty hard lately. There's nothing inherently wrong with that, but praising XMPP without mentioning its drawbacks, while spreading outdated and vague criticism of other options, is a somewhat misleading way to do it... and a disservice to the community.