If there is abuse from a mod or an admin, you go somewhere else - either join another similar community or server or make your own.

Or in the case of abuse from mods, report them to admins (assuming admins are reasonable).

Remember, this isn't reddit. You don't have to live with the mods or admins, you can just stop going to that instance/community and find somewhere else.

Also ultimately being able to view votes is a way to combat abuse, as you can spot people who just down vote someone else constantly.

EDIT: Honestly confused about the down votes, someone would care to elaborate? :)