According to the documentary Hell on Wheels it was mostly built by a man named Cullen Bohannon with his own hands. When he got tired, he used some Chinamen and I think maybe a woman dressed as a man for a stretch.