well yeah if you cherrypick a two sentence synopsis you can make anything sound ridiculous.
https://en.m.wikipedia.org/wiki/Criticism_of_evolutionary_psychology
...criticisms include disputes about the testability of evolutionary hypotheses, cognitive assumptions such as massive modularity, vagueness stemming from assumptions about the environment that leads to evolutionary adaptation, the importance of non-genetic and non-adaptive explanations, as well as political and ethical issues in the field itself.
those are all pretty significant criticisms.
regarding the racism specifically, you need to read between the lines. of course they're not going to outright admit they are being racist. But when you are dealing with unfalsifiable/non-empiracle hypotheses, while over-emphasising biology (race/sex), that's not science, that's politics wrapped in a scientific facade.