You don't have to be subscribed to see activities in a community (including downvotes) but someone on you instance does for the community know to send the activities to that instance.
Disabling downvotes doesn't actually prevent items being downvoted, it just hides them from those that don't want to see it.
Disabling donvotes in the instance setting does prevent them being recorded on that instance. You can see the code here.