Look, if you run the server you have access to metadata of clients connecting to it. That is networking 101. And that Signal shares phone numbers and connection timestamps is well established by court documents.
The security audits are of the code and encryption algorithm, not the infrastructure.