Signal gets me all the privacy I need. I don't care if they know my phone number uses Signal, I don't use it as anonymous chat, I use it with friends and family.
As others in this post have said, Signal handles privacy perfectly fine, it does not provide anonymity.
Unlike several other users here, I actually view Signal's contact discoverability as a feature, not a security flaw. All it means is if someone I know installs Signal, they can easily send me a message without a complicated back and forth through some other medium.