Some of the time it is though. Like Gmail has a pretty large list of IPs it won't deliver email from. When self-hosting, it's something you really do have to worry about.
The reason most people don't worry about it is that most people only use a handful of free emails and organizations that provide email addresses for their users spend time worrying about it so users don't have to.