I honestly do this too (upvote almost everything, but I try to avoid anything that might be problematic) but I do think people would (hopefully) take into account accidents when looking at a user’s overall history.
That’s a good question, actually…. Are the visible upvotes per post or could you go to a persons account and view everything? I’d kind prefer the latter just to avoid jumping to conclusions if someone does accidentally upvote something bad.