It seems a downvote on a post affects your reputation negatively, while an upvote does not. Boosting a post and/or comment increases your reputation score instead. The scheme is weird. I posted a video a couple days ago that had over a dozen upvotes but 2 downvotes. My reputation score was -2.

I realize they're internet points, but I went ahead and boosted all my comments since kbin doesn't do it automatically yet. My reputation score was in the teens after that. It's a work in progress system, and already known. Apparently a fix is under way.