We actually decided in like the 50s or 60s that it does matter.

As someone else already mentioned, sex traffickers aren't a protected class.

The government should not force someone under threat of violence to do work.

So if there's a feedlot set up next door to your house, the government shouldn't proactively require them not to dump the pig slurry into your backyard? Sure, maybe you can sue, but they're thousands of times richer than you, and as the suit goes through the courts, you're still buried in pig shit.

How about every credit card company refusing to deal with you because, for example, you're a libertarian? Just fine, start your own bank? And how about a water company or some other lifeline utility?

It's entirely reasonable for governments to impose standards on business. Everywhere and every time that has not happened, businesses have committed abuses. Even Adam Smith (the real one, not the modern crackpot who stole his name) knew that. "People of the same trade seldom meet together, even for merriment and diversion, but the conversation ends in a conspiracy against the publick, or in some contrivance to raise prices."