There is literally no way you will build a decent modern house without thousands of man-hours, only in the construction itself. That's before you count the hours needed for getting the materials you need.
It's unreasonable to posit that we should design our society such that you need to save up enough money to finance something like that up-front in order to build a house. Why not go for the (current) solution, where I can loan money to finance the house, then pay that back once I have stable living conditions (because I now have a house)?