I haven't tried making .XCompose work on wayland. But if you use Plasma as a desktop, it does have an option to set up the compose key. I never checked if it let you create your own compose "recipes". I also don't know how it handles it in the background. But if you set it up through Plasma, the default compose behavior works flawlessly both on X11 and Wayland.