Because the majority of people not having babies don't plan to have babies, and the majority of people who want babies still manage to have them, though it might take more attempts than it used to, or IVF assistance. The desire and cultural influences are much bigger factors at the moment on birth rate than endocrine disruptors.

(Though they're probably responsible for a bunch of other stuff, like the obesity epidemic)