Remember metadata , they also know person A send message to person B at specific time. Of course knowing phone number also gives them your location from cellular towers.