Lots of people browsing all local communities probably downvoted posts in communities they weren't members of. In especially more niche communities this can mean that the downvotes from people who aren't in it can drown out the upvotes from people who are. Allowing only upvoting means that the number reflects who many people actually actively liked it
Edit: ah sorry, just read the comment from @justsillyme@lemmynsfw.com below and they said pretty much the same thing I did but better.