In their defense, if everyone was perfect everything would be perfect.

Right, but fascism could not exist in that kind of world because it's based, at least partially, on exploiting inequalities to turn citizens against an "other" (usually already-marginalized groups at first). So if "everyone was perfect," there would never be the right conditions for fascism to take root.