They think that "racism" is when you're openly lynching brown people in the town square. Or marching in klan robes. Or Nazi saluting a Nazi flag. Because they aren't doing that, it means they're not racist. They're not smart enough to understand anything more subtle or indirect like generational poverty caused by a series of intentional hamstringings from enslavement to redlining and the like.

I know a lot of people who voted for Trump the first time because "he says what everyone is thinking," but when pressed, they would only reluctantly admit that they liked what he said about foreigners and minorities. But, of course, they're "not racist or anything."