Fascists absolutely care who you voted for. They put party (and then country) above everything else.
The word you were looking for is racists. They care about skull sizes and skin color.
These groups are not mutually inclusive, but there is a whole lot of overlap.
