I've read Settlers and Fanon (and The Counter-Revolution of 1776, and I'm working on Black Reconstruction in America right now).
comparing a random Black supremacist, someone on the social, political, and economic fringe, with the deeply entrenched and omnipresent system of colonialism is utterly ludicrous
Besides reactionaries and overt racists, no one is claiming there is hegemonic, systemic, or structural racism against white people. I've never even heard this from libs, let alone leftists. Instead, the argument is that insulting or harming someone based on the color of their skin still fits the definition of racism, even if it is not aligned with a larger superstructure of racism.
I'm convinced this whole discourse could be resolved with "there is no structural racism against white people, even though individuals may be racist against them." The disagreement stems from arguing that only structural racism is really racism, which strikes me as an odd semantic battle to pick.