Fluxer (more like discord) or Matrix

matrix

Mattermost

Because it hasn't been mentioned:

IRC

Sometime also mentioned SimpleX, which I'm looking into myself, but have no experience currently.

For one to one chats and the occasional group chat? XMPP is pretty awesome and has lots of good clients (Conversations on Android, Monal on iDevices, Dino, Gajim, etc. on Linux)

If you want something closer to Discord and host public communities or very large chatrooms, Matrix is very good. You can self-host it with Continuwuity, and there are lots of clients too (Cinny, FluffyChat, Element, Fractal, NeoChat, etc.)

If you just need voice chat, Mumble is simple and works well.

Matrix, and I recommend selfhosting Continuwuity as it is lightweight and easy to spin up. Your other option could be XMPP with Snikket or proper Prosody.

Though do you have any particular features you wanna bring over from Facebook Messenger?

Matrix

https://delta.chat/

Are you asking just for general self-hosted messengers? Why did you mention Facebook specifically?

Nextcloud Talk

Matrix and element clients

simplex

I tried a bunch. Right now I'm running Snikket (xmpp) and it's been very reliable.

Pidgin, maybe. I like it.

Edit: Oops, I might have misread/misunderstood what you were asking. If you're asking about server software, I'd suggest XMPP :)

Though I know Matrix is very popular (as shown by people others suggesting it)