Not necessarily. It depends on what you're optimizing, the impact of the optimizations, the code complexity tradeoffs, and what your goal is.
Optimizing many tiny pieces of a compiler by 0.02% each? It adds up.
Optimizing a function called in an O(n^2^) algorithm by 0.02%? That will be a lot more beneficial than optimizing a function called only once.
Optimizing some high-level function by dropping into hand-written assembly? No. Just no.