You need the web site to use a certificate from the same root authority as your client certificate.

I'm not sure if I've misunderstood you, but I use Lets Encrypt for the server's TLS, and then my own CA cert (which is only present on the webserver) for the client's mTLS and everything works fine, since it's the client that validates the server's cert and the server that validates the client's cert.