Booms have market opportunity, and busts have market opportunity.
That said, having too many people to do the work is a lot less dangerous than not enough people to do the work.
The bust comes with a high risk of a total economic collapse. Those houses get cheap, but food, sanitary, and energy security are at risk. Living in a rotting infrastructure is very different than living in an expanding but underdeveloped infrastructure.
Leveling out or a slow population decline would be good in some scenarios. If we're going from 2.5:1 down to 0.25:1, you'll end up with another boomer wave, pissed off at the government, ripe for scenarios that make today's problems seem like an annoying party.