You're right, but security and privacy is about layers, not always 100% effective mitigations, especially not when the mitigation is a function (contact discovery) that requires a private list (your contacts) be compared against another one. For anyone where this is an actual security risk, they don't have to to share their contacts. They will not know which of their friends/family are on Signal, but they can still use the service.

This feature does protect users in that any legal court order for Signal to present who is friends with who (as almost every other messaging provider has actual access to your list of contacts) is not possible. They've been subpoenaed multiple times[0] and all they can show is when an account was created and the last day (not time) a client pinged their servers.

Lastly, I'm not sure if this is even a feature or not but it wouldn't be too difficult to introduce rate-limiting to mitigate this issue even more. As an example, its very unlikely that most people have thousands (or even tens of thousands) of people in their contacts. Assuming we go just a step beyond the 99th percentile, you can effectively block anyone as soon as they start trying to crawl the entire phone number address space, preventing the issue you're describing.

[0] https://signal.org/bigbrother/