Setting up is a piece of cake but getting your emails through spam filters can be a pain.

Have you considered Tuta?

I don't know any guide, but deliverability is the biggest concern and it may also not be fully under control. Sometimes IP blocks get blacklisted or deranked and your emails end in spam, and you might not even know that.

I think the main difficulty is getting companies like Gmail to recognize your domain as legitimate, as they don't by default

I don't think I do this like you're suggesting, but I have my email hosted at opalstack. I've been really happy with them. I don't have a server-side spam solution yet, though. I just set up spam rules on Thunderbird on my local machine.

You're gonna need a few books and a full time engineer

If you self host it, do it on a VPS so if you're out of town and it goes down, you don't lose a month's worth of emails