No one is trying to 'make it a race thing'. It was already 'a race thing' and we are actively trying to un-race it.
You and I being unaware of the partially racial origins of these terms (yes, I just learned about this too), doesn't erase the implications of these words and how it can be hurtful to some people.
There's no such thing as anti-wokeness, only willful ignorance. We can be better people than that.