Luckily, here, while admins can see how many up- and downvotes you have, it doesn't influence anything, and that's by design; besides, it's unlikely that any proper implementation of karma system is possible in a federated network for individual users.