But they were exceptions, not the majority who were slaves/workers. You'd still expect a heavy selection bias for good body attributes if you sampled them at random, assuming OP's hypothesis is true.