This is the problem we have to overcome.
Look at the old time abolitionists. People today try to paint them all as morally driven saints. The fact is that a lot of the opposition to slavery came from Northern capitalists who were mad they couldn't have slaves in their factories. Other opponents were tradesmen who couldn't compete with the slaveowners who rented out their slave carpenters and blacksmiths.
Pragmatism gets shit done.