It matters to connect with the right people, that’s all.

^This right here is the key, especially here. If there's a post that only has three upvotes it isn't a "bad" post like the downvote people want to say - it's niche. So in that case, where the audience for something is so few people then the downvoters could prevent those three people from seeing the niche content that they like. Selfish, really.