Check perms on the ~/.ssh/authorized_keys file on the server side (should be 0600) and the ~/.ssh dir itself?