My client keeps a reputation counter totaling the up and downvotes someone receives from me. It tends to be asshole behavior that draws a downvote from me, so if someone manages to rack up a significantly negative total (I've never seen anyone hit even -10) that's a good sign to just block them.

Other than that, nothing and it's rare I bother with them.