You're right, and even the Lemmy devs get this wrong in the docs:

You can upvote posts that you like so that more users will see them, or downvote posts so that they are less likely to be seen.

Note that it doesn't talk about the quality or appropriateness of the comment, just that you can suppress it by downvoting.