Except that wasn't a serial port, it was midi, and the reason it was on the sound card was because the input was analog.
Your joystick was just two fancy potentiometers, and your soundcard decoded the voltage on the middle legs into a position.
Soundcards handled joysticks because they had the fastest ADCs.