In a single one-off program or something that's already fast enough to not take more than a few seconds—yeah, the time is spent better elsewhere.

I did mention for a compiler, specifically, though. They're CPU bottlenecked with a huge number of people or CI build agents waiting for it to run, which makes it a good candidate for squeezing extra performance out in places where it doesn't impact maintainability. 0.02% here, 0.15% there, etc etc, and even a 1% total improvement is still a couple extra seconds of not sitting around and waiting per Jenkins build.

Also keep in mind that adding features or making large changes to a compiler is likely bottlenecked by bureaucracy and committee, so there's not much else to do.