To me this seems like a routing issue. Some things to check:
- Can a port checking service (like canyouseeme) see 443?
- Can you connect to 443 from the internet (use a VPN or separate network)?
- Can you edit your hosts file to point
jellyfin.mydomain.comto your jellyfin server's LAN IP? - After doing this, can you connect to
jellyfin.mydomain.comon the LAN?
My guess is the router is routing traffic to it's external IP from the LAN back to itself, without following port forwarding rules. Good luck figuring it out though!