I'm working on it. Maybe. Well, one day. Like when other Left of centre, anti trump, anti corpo type folks oppose the things I oppose in correct manner and degree, duh.
I have had a very similar observation based on a feature of Voyager that shows you the total upvotes you have given a user, net of downvotes, next to their name.
In addition to what you observe, I’ve also saved myself from being too dismissive or reading a comment more carefully based on that number at least being an indication that I’ve had good interactions with this person in the past.