Well if Void Linux is unpopular enough for it to be a problem, are the Gentoo-based ones better in that regard? Like, does it have comparable repositories to gentoo?
Gentoo does support binary repos, I haven't researched them in depth, but creating a binary-only child distribution should be very possible.
Except as you said - I don't think is popular enough, I don't know how comprehensive the current set of binary packages is.
But it should mostly be just a matter of actually precompiling everything with a reasonable config