"Attention is a limited resource"
What units do you measure that in? Minutes/hours? Everyone has the same amount of it each day, and everyone rations their own attention however they see fit. One person spending hours on the fediverse and upvoting hundreds of posts isn't taking anything away from someone who spends minutes on it and upvotes a dozen.
Whatever you mean by "silent majority", it doesn't matter. You're missing the point that nothing is being taken away from them when someone else votes more than they do.