Easiest and most affordable is probably a security key like the Nitrokey or the https://www.yubico.com/. I personally don't like the company behind yubikey much, but if you want something small you can always leave in the device that's pretty much your only option.

For "cheaper, but a bit more effort" would be just getting a smartcard blank, a card reader (if you're not lucky enough to have a notebook or computer with one built in), and then either write your own applet, or use one of the available opensource ones, and upload it to the card. A variant of that would be the Fidesmo card, where you get a card and their applet.

Or you just use the TPM you may have in your system - though you'll need to be careful with that: Typically one reason for using a hardware token is to make sure keys can't get extracted, while TPMs often do allow key extraction. Software to make that work would be opencryptoki.

Generally you'd use PKCS#11 to have the various components talk to each other. On your average Linux pretty much everything but GnuPG place nice. with PKCS#11. Typically you end up with pcscd to interface with the smartcard (the above USB tokens are technically also just USB smartcards), OpenSC as layer to provide PKCS#11 on top, and software (like OpenSSH) then talks to that.

All of that should be available as packages in any Linux distribution nowadays - and typically will also provide p11-kit configured to use a proxy library to make multiple token sources easily available, and avoid blocking on concurrent access.

ssh-add supports adding keys from pkcs#11 providers to the SSH agent (search pkcs11 in ssh-add manpage), with some distribution (like RedHat) also carrying patches allowing you to only select individual tokens for adding.

If you're also using GnuPG it gets more complicated - you pretty much have two options: Stick with PKCS#11, in which case you'd replace GPGs own smartcard agent with gnupg-pkcs11-scd, or you use GPGs own card implementation, in which case you can forget pretty much everything I wrote above, and just follow the security key manual for setting up a GPG card, enable SSH agent support in the GPG agent, and just use that for SSH authentication.