Void offers musl too. Unless they've discontinued it.

But

compile everything yourself?

I do (almost) exactly that. I run Gentoo almost everywhere. The 'almost' is because Gentoo now offers an official bin repository too, so I can mix compiled and pre-compiled software. (Although you've always had the option to set up your own binary host).