If you're concerned about privacy, you should probably know that a lot in the fediverse is very open by design so that other instances can communicate effectively.
That's why everyone can see what you upvote/downvote, even if Lemmy itself doesn't show this to you in its UI.
No, I've actually never had phrases get stuck in my head. Nowhere near the way songs do at least.