Well... you say that.

That kind of dynamic is basically what existed in the antebellum South, and it didn't exactly get in the way of maintaining white supremacy (racial purity has always been socially constructed anyway). White men raping black slaves was very common, to the point that it was more or less an expectation, at least among the wealthy. This was the twisted logic behind the norm of "women should remain pure, but men should have experience," which has carried over to the modern day even though it doesn't make sense without an underclass of sex slaves.

Whether through knowing the history or through following the same logic to the same conclusions, that's what reactionaries aim to recreate.