but in that chain what you really care about is your phone number that identifies you in the real world to your messages, right?
It doesn't matter, what matters is that the server has a unique id for you and the person you're talking to, and that id can then be mapped to the phone number that was initially collected. That's all the server needs to identify the real identity of the people you communicate with.
It's not a question of what the server needs minimally, it's a question of what the server could be doing if it was set up maliciously. The sealed sender does not solve this problem in any way shape of form.