You only have to optimize the factory enough to protect yourself. If you can progress faster than the biters, you're probably fine.