It matters to them, so I guess you are not the type of user they want. It's annoying, but if they want their server to be more exclusive, they can do that. It is the beauty and the pain of federation.