Implementing E2E isn't just about the encryption though, it's also about the key exchange/distribution/generation approach.

If you look at what Matrix does, so much of the complexity comes from how they authenticate all the different clients to manage E2E in a distributed way. For proper E2E you've got more than 2 ends (multiple clients) so you need to manage it for all.