We had the same problem after the civil war.
The slavers were kept out of power for 10 years, with excellent results.
Then political priorities shifted and they were allowed back into power.
Cue the KKK, Jim Crow, and the resumption of 'Southern Culture' (ie evil racial brutality).
The nazis were largely displaced for decades, the slaver class wasn't, and it destroyed America.