Yeah no, we need density.
You know what those small towns you're thinking of are? Urban sprawl. The entire town is urban sprawl. No towers. Single family detached homes everywhere. No good public transit because there's not enough density to support it. Rentals are 80% basements.
Moving more people into those towns without pushing for densification is going to lead too ... You guessed it, even more urban sprawl!
Is that really the path you think we should be going down?