It's a myth.

Yes it takes longer, but specialy on headless server updates are pretty fast

Big boys like LibreOffice Firefox have also pre built binaries if you so prefer as well ..

I use Gentoo since amd k6-400 MHz times so today build times feel like no wait at all