You can use a reverse proxy for that. Start multiple Gemini servers listening on localhost:19650, localhost:19651, ..., and have the rproxy listen on *:1965. Then configure the reverse proxy to redirect to the correct port depending on the TLS SNI (hostname) requested.
That, or use a Gemini server that supports virtual hosts natively :)