I have my own ssh server (on raspberry pi 5, Ubuntu Server 23) but when I try to connect from my PC using key authentication (having password disabled), I get a blank screen. A blinking cursor.

However, once I enter the command eval "$(ssh-agent -s)" and try ssh again, I successfully login after entering my passphrase. I don't want to issue this command every time. Is that possible?

This does not occur when I have password enabled on the ssh server. Also, ideally, I want to enter my passphrase EVERYTIME I connect to my server, so ideally I don't want it to be stored in cache or something. I want the passphrase to be a lil' password so that other people can't accidentally connect to my server when they use my PC.