My theory is that you already have something providing ssh agent service
in the past some xserver environments started an ssh-agent for you just in case of, and for some reason i don't remember that was annoying and i disabled it to start my agent in my shell environment as i wanted it.
also a possibility is tharlt there are other agents like the gpg-agent that afaik also handles ssh keys.
but i would also look into $HOME/.ssh/config if there was something configured that matches the hostname, ip, or with wildcards* parts of it, that could interfere with key selection as the .ssh/id_rsa key should IMHO always be tried if key auth is possible and no (matching) key is known to the ssh process, that is unless there already is something configured...
not sure if a system-wide /etc/ssh/ssh_config would interfere there too, maybe have a look there too. as this behaviour seems a bit unexpected if not configured specially to do so.