You can't aggregate them internally, anyway. You need to be able to know if someone already voted on something.

I think activitypub needs to be extended so that the likes and reduces only need to be sent to the host of the content, with federation then being told just the aggregate number. Then the only servers that need to know identity of votes are the host server (necessary to ensure nobody can multi vote) and optionally the server the user voted on (could just relay the information to the host server and not store it locally, but then it'd be harder to tell what you've already upvoted -- could use local storage but I think lots of people use social media on multiple devices).