Is your point seriously that the people who aren't active don't influence the the fediverse as much as people who are?
That's really weird, and if you just think about that for a minute you should see why that's weird.
Like, imagine there's an election and the board of elections was suddenly like "welp, according to this chart, a lot of registered voters don't regularly vote, so it's unfair to let some voters who vote in every election have a disproportionate impact on the election results. Therefore, we're canceling voting."