I strongly encourage you to strive for more precise language. If observers regularly make the same rhetorical rebuttals to your statements, that's a signal that you should seek a clearer phrasing.
When you say something like "misandry isn't a thing" or "blacks can't be racist towards whites", it's very easy for someone not already familiar with modern usage to hear "[hating men because they're men] isn't a thing" or "blacks can't be [racially biased against] whites", even if you meant "[sexist oppression of men] isn't a thing" or "blacks can't be [systematically advantaged over] whites." And then we all get to waste time arguing over the precise meaning of words while those in favor of oppression gleefully watch.
Likely every white guy you've ever met has a first or second had experience of being presumed dangerous because they're a man or being presumed a bigot because of their skin color. These presumptions are often soft and quickly discarded, and they certainly aren't a big a problem as rape-culture or the school-to-prison pipeline, but just because the wrongs are lesser does not mean they don't exist.