Very impressive work
I use Nautical. It will stop your containers before performing an RSYNC backup.
Yes the containers do need to be stopped. I actually built a project that does exactly that.
If you just need to backup Docker Volumes, I recommend Nautical.
You can use it to backup to an NFS share if you need to go between Machines.
Proxmox on bare metal. Then a TrueNas VM for storage. And a Ubuntu VM for containers.
I'm a little biased because I built it, but this is what I use.
I create backups of my Docker compose files and each container volume (these are done when the containers are stopped so no data is corrupted). Then I take those folders and send them encrypted to Backblaze using Kopia. Since it's just config files I am able to get away with the Free tier.
I also have my Docker compose files backed up in a private GitHub repo.