I think you might be underestimating how personal these numbers can be to some people. I'm glad they don't impact you, but many people, especially the upcoming generation, equate these numbers with their value.
Big numbers can make people feel validated, that their opinions are valued, or they're funny.
Negative numbers may result in disappointment or feelings of rejection.
I don't think negative people are "just negative". Toxicity pervades cultures which allow it to spread. Down voting is sometimes enough to act as a nucleation point. I've seen heated arguments start over accusations of down voting, which isn't isolated to their thread.