That's great but I don't believe that changes my overarching point.
Interestingly by the core tenents of Libertarianism, segregation to colored bathrooms would still be alive and well, since private property and individual freedom reign supreme. There'd be a similar recognition or lack thereof for things like same-sex marriage or Trans rights.
Taken to its logical conclusion, such a libertarian utopia would be a hodgepodge of private properties with arbitrary and often draconian local laws dictated by landlords. In effect it would probably devolve into something of a feudal system.
Can't simultaneously have a small toothless government and at the same time one with enough authority to ensure equal rights and counteract discrimination. There's a reason the biggest proponents of Individual Freedom and states rights tend to be the most backwards, socially.