Those old spinning disk drives are notoriously slow. You’d be better off with a USB flash drive in some cases.
But your best bet is biting the bullet and getting an SSD. They’re not terribly expensive anymore. You’ll pay about as much as a game costs for one, whether you want an external one to connect via USB or get the internal SSD and make the upgrade to a larger drive.
I haven’t done this on the Series X but I did do this on the One X and it was fairly easy following Microsoft’s guide online. I recently bought a 2TB SSD for $130 at Best Buy to upgrade in my gaming computer at home. This is the same kind that goes into your Xbox Series X internally. You could also spend about double and go for the 4TB option.