It sounds pretty reasonable. As long as you keep SSH patched and keep the key safe it should be quite locked down. Do double-check that password login isn't allowed (or that all users have a very strong password).
One non-security note is be careful with rsync backup. Generally rsync isn't considered a backup as any mistakes made in the source will be propagated to the "backup" on next sync. Although there are ways to use rsync to take good backups (like copying to a new directory for each backup).