If you're not making posts that the community members want to see, then they are justified in downvoting you so that they do their part to influence the culture of the community.
The fact that you're only targeting downvoters and not upvoters leads me to believe you want to surround yourself with yes-men instead of authentic humans.
If you think 5 people downvoting you is what's stopping your posts from gaining visibility, then you are simply incorrect. Lemmy is still very small and most communities that aren't tech or politics aren't "visible" because they lack activity.
Can you make your argument without resorting to inaccurate analogies? That way we can focus on the topic instead of getting derailed.