Inbreeding is more likely to produce offspring, but also more likely to produce offspring with genetic defects, and marrying someone very genetically different is less likely to produce offspring, but dangers are fewer.

I mean, provided you can't find a neanderthal to marry today, the latter concern is not too important.

So both are not too bad, fertility is important sometimes. It's the natural regulation of population.

BTW, about diversity and all that ... it's good, but triggers the psychological and biological mechanisms (EDIT: reducing fertility) in the parts of population smart enough to be diverse.

So nature also regulates such disturbances. Like the Force in Star Wars. Humanity works the way that if smartasses become too smart, dumbasses would have their asses so it would all cancel out to a more even distribution.