Similar but slightly different reply to Kelly's, not really responding to your comment on testing viability (though I don't feel like this comment should be a top-level comment).
It seems to me that at least for opt=size, it actually did improve performance (for a basic benchmark at least) somewhat for low cost compared to no flags. I'm sure this is not as fast as opt=speed, but I would call that more of a trade-off than a sacrifice especially when also considering compilation-time to get a measure of efficiency.
At least that was my impression with Clang (which I was seeing size-optimized Clang giving decent performance but half the compilation time of default GCC). An efficient middle-ground.
EDIT: Though this was also for code (via bindings) not Godot/exports themselves, so it could be different (though I'm not sure why unless a difference of defaults).