Yeah and I agree with that.
The issue comes in when actually trying to implement affirmative action. It will, sometimes, temporarily demand discrimination to be done to correct for past injustices. And that discrimination is sometimes going to be based on race.
But that discrimination based on race is what a lot of people are calling "racism". But it is not the same as actual racism. Not in effect or in principle.