You need to use your imagination more. A robotic construction site will not look anything like a human one. Bags of readymix concrete have the form they do because they are intended to be carried by a human being. But the goal isn't to get a bag from point A to point B. The goal is to get the concrete from point A to point B. Instead of a humanoid robot carrying a bag of readymix, imagine an automated hopper on the end of a crane, with a computer vision and reasoning system that allows it to precisely place it where it needs to go. And instead of that concrete being formed by a humanoid robot holding a trowel in its hands, it will be mounted on a robotic arm mounted on a glorified roomba.

When we automated car factories, we didn't do so by making millions of mechanical men. Construction will be no different. Individual humans are not masters of every construction trade. Why should we demand the robots be masters of all trades? We can simply purpose-build robots for particular construction tasks. Builders wealthy enough may own a fleet of them. Smaller operations will rent them as needed. Just replace the cost of hiring subcontractors with the cost of renting a particular trade robot.