I think you can call someone saying “it’s unfair to judge people by race” a racist when they're using that line to applaud the removal of protections against institutional racism. We can argue the merits of AA as a form of protection, but it was protection nonetheless. To say that it was unfair is to entirely ignore the unfairness which necessitated its existence.