No, you're getting confused by the fact that a large percentage increase in a small number is still a small number. And that the healthiest outcomes have an ideal spot at the late 20's to begin with, and that having babies before 25 has slightly elevated risk of poor outcomes, as well.
Down syndrome risk peaks at around maternal age 44, but with about 0.6% of pregnancies. That's still a small number.
Other concerns like preterm birth and low birth weight tend to have the healthiest outcomes at maternal age in late 20's. For example, studies like this show that preterm birth is less likely for 30-32 year olds than the broad category of "younger than 30," and low birth weight for gestational age is lower between 30-40 than it is for younger than 30.
Either way, though, the correlation is so weak and the overall odds are so low that it's like playing with slightly different weighted dice. In the end, maybe some statistical significance might appear across a huge sample size, but for individuals making their own family planning decisions, the probability differentials are too small to place too much weight in those stats. Statistical significance isn't real-world significance.
Throw in other advantages of delaying birth, like those who defer for educational/career reasons and find themselves in a different demographic category (wealthier, less likely to be divorced) as a result, and it might actually be advantageous for a 20-year-old's future children to intentionally pursue paths that lead to later births.
Just let people do what they want with family planning: have kids, don't have kids, have kids young, have kids old, have lots of kids, have fewer kids, adopt, IVF, surrogates, whatever. These are deeply personal decisions that people should be left alone on.