Two things are true at once:
- no people is Semitic (because the word applies to languages not to peoples)
- antisemitism is the hostility towards Jews as Jews (because words have meaning beyond their etymology, just like homophobia does not imply being scared of homosexuals)
Otherwise, what, antisemism is the hostility towards speakers of Semitic languages? That's just silly.