Obfuscation is meaningless. It's public info or it's not. In this case it's necessarily public

It's not really possible to have an upvote counter without storing who voted, so your instance admin would always know

If there is abuse from a mod or an admin, you go somewhere else - either join another similar community or server or make your own.

Or in the case of abuse from mods, report them to admins (assuming admins are reasonable).

Remember, this isn't reddit. You don't have to live with the mods or admins, you can just stop going to that instance/community and find somewhere else.

Also ultimately being able to view votes is a way to combat abuse, as you can spot people who just down vote someone else constantly.

EDIT: Honestly confused about the down votes, someone would care to elaborate? :)