Why would I build affordable housing? That just invites the poor. /s

There are a bunch of new apartments in my city's downtown (USA). They're all premium spaces. I can afford a spot, but I doubt most can.

I agree that's part of the problem, but see my comment below. Stats show we have an all time low for housing:people compared to our past and compared to most other western countries.

To fix it, for sure we need scaling property tax rates and higher empty/vacant housing taxes, but my point is that even if we forcibly removed 2nd or 3rd houses from every single person/corporation, as well as taking any empty/vacant housing, and distributed it, we still wouldn't have enough to be on par with our historical rates OR other western nations