Downvotes are an inherently unequal proposition, as they are now implemented. This allows everything from near and dear friends who respectfully disagree to randos with day-one accounts who don't even know what a community is all about, to brigading events organized in a larger community (possibly on Reddit or in Matrix or Discord or such). e.g. iirc I can user-block someone or even an entire instance, but in retaliation they can see my profile and downvote everything I have ever done, or have a bot do so within seconds of new material coming out. Which would affect its discoverability.
Potential solutions would be to make them no longer anonymous, and/or when you block a user or an instance then they can no longer downvote that content - just like a user-level defederation. As it is now, user-level blocks are extremely weak and even notifications can be delivered by simply tagging someone's username.