I still have it from time to time that Windows has to install a driver for something benign like a thumb drive. Not always, though. And yes, the driver is fixed to the physical port. Using a different port reinstalls the same driver again.

Experienced this exact behavior on Windows Vista, Windows 7, Windows 8, Windows 10 and Windows 11.