Looking for help: Running the same commands for multiple, similar variables (lemmy.dbzer0.com)

submitted 1 month ago by [email protected] to c/[email protected]

7 comments fedilink hide all child comments

For fun, convenience and a chance to learn shell scripting for myself, I've been working on a collection of scripts to semi-automate the backup of my computer, two network connected Raspberry Pi's and my Android phone with Termux.

The main script basically runs a bunch of remote backup scripts, then copies those remote backups to a dedicated partition on my computer and finally creates a copy of that partition to an external drive connected to my computer. I use rsync and it's dry run feature so I am able to examine any file changes which has been super useful to me for catching mistakes and issues as I've been learning how to self-host over the past half year.

I have a simplified version of those backup scripts that makes a copy of my /home directory:

#!/bin/sh



# VARIABLES
## ENABLE FILE TRANSFER: Change variable `DRY_RUN` to `#DRY_RUN` to enable file tranfers
DRY_RUN="--dry-run" # Disables all file transfers

## PATHS (USER DEFINED)
SOURCE_DIRECTORY_PATH="/home"
SOURCE_DIRECTORY_PATH_EXCLUSIONS="--exclude=lost+found --exclude=.cache/*"
BACKUP_NAME="home"
BACKUP_BASE_PATH="/backup"

## PATHS (SCRIPT DEFINED/DO NOT TOUCH)
SOURCE_DIR="${SOURCE_DIRECTORY_PATH}/"
DESTINATION_DIR="${BACKUP_BASE_PATH}/${BACKUP_NAME}/"

## EXCLUSIONS (SCRIPT DEFINED/DO NOT TOUCH)
EXCLUDE_DIR="${SOURCE_DIRECTORY_PATH_EXCLUSIONS}"

## OPTIONS (SCRIPT DEFINED/DO NOT TOUCH)
OPTIONS="--archive --acls --one-file-system --xattrs --hard-links --sparse --verbose --human-readable --partial --progress --compress"
OPTIONS_EXTRA="--delete --numeric-ids"



# FUNCTIONS
## SPACER
SPACER() {
    printf "\n\n\n\n\n"
}

## RSYNC ERROR WARNINGS
ERROR_WARNINGS() {
    if [ "$RSYNC_STATUS" -eq 0 ]; then

        # SUCCESSFUL
        printf "\nSync successful"
        printf "\nExit status(0): %s\n" "$RSYNC_STATUS"

    else
        # ERRORS OCCURED
        printf "\nSome error occurred"
        printf "\nExit status(0): %s\n" "$RSYNC_STATUS"
    fi
}

## CONFIRMATION (YES/NO)
CONFIRM_YESNO() {
    while true; do
        prompt="${1}"
        printf "%s (Yes/No): " "${prompt}" >&2 # FUNCTION CALL REQUIRES TEXT PROMPT ARGUMENT
        read -r reply
        case $reply in
            [Yy]* ) return 0;; # YES
            [Nn]* ) return 1;; # NO
            * ) printf "Options: y / n\n";;
        esac
    done
}



##### START

# CHECK FOR ROOT
if ! [ "$(id -u)" = 0 ]; then

    # EXIT WITH NO ACTIONS TAKEN
    printf "\nRoot access required\n\n"
    return

else
    printf "\nStarting backup process..."

    # ${SOURCE_DIR} TO ${DESTINATION_DIR} DRY RUN
    SPACER
    printf "\nStarting %s dry run\n" "${SOURCE_DIR}"
    rsync --dry-run ${OPTIONS} ${OPTIONS_EXTRA} ${EXCLUDE_DIR} "${SOURCE_DIR}" "${DESTINATION_DIR}"
    RSYNC_STATUS=$?
    ERROR_WARNINGS

    # CONFIRM ${SOURCE_DIR} TO ${DESTINATION_DIR} BACKUP
    SPACER
    if CONFIRM_YESNO "Proceed with ${SOURCE_DIR} backup?"; then

        # CONTINUE ${SOURCE_DIR} TO ${DESTINATION_DIR} BACKUP & EXIT
        printf "\nContinuing %s backup\n" "${SOURCE_DIR}"
        rsync ${DRY_RUN} ${OPTIONS} ${OPTIONS_EXTRA} ${EXCLUDE_DIR} "${SOURCE_DIR}" "${DESTINATION_DIR}"
        RSYNC_STATUS=$?
        ERROR_WARNINGS

        printf "\n%s backup completed\n\n" "${SOURCE_DIR}"
        return

    else
        # SKIP ${SOURCE_DIR} TO ${DESTINATION_DIR} BACKUP & EXIT
        printf "\n%s backup skipped\n\n" "${SOURCE_DIR}"
        return
    fi
fi

##### FINISH

I would like to adapt this script so that I can add multiple copies of the following variables:

## PATHS (USER DEFINED)
SOURCE_DIRECTORY_PATH="/home"
SOURCE_DIRECTORY_PATH_EXCLUSIONS="--exclude=lost+found --exclude=.cache/*"
BACKUP_NAME="home"
BACKUP_BASE_PATH="/backup"

without having to make multiple copies of the following commands within the running script:

    # ${SOURCE_DIR} TO ${DESTINATION_DIR} DRY RUN
    SPACER
    printf "\nStarting %s dry run\n" "${SOURCE_DIR}"
    rsync --dry-run ${OPTIONS} ${OPTIONS_EXTRA} ${EXCLUDE_DIR} "${SOURCE_DIR}" "${DESTINATION_DIR}"
    RSYNC_STATUS=$?
    ERROR_WARNINGS

I'm mainly just looking for a way to avoid touching the script commands itself so I don't have to change the variable names for each additional directory I want to add. I'm not sure what that would be called or where to look. Any help would be greatly appreciated.

you are viewing a single comment's thread
view the rest of the comments

[-] [email protected] 2 points 1 month ago

try storing the paths to backup in a dot-file; one path per line. then, in the script, iterate that file line by line backing up the paths.

[-] [email protected] 1 points 3 weeks ago* (last edited 3 weeks ago)

I ended up using dot-files like you suggested. Instead of multiple source/destination locations within one dot-file, I've set it up to read multiple dot-files that contains one source/destination location each.

Now I can use the command dir-sync /path/to/directory/.sync1 /path/to/directory/.sync2 or dir-sync /path/to/directory/* which is close enough to how I imagined it working.

The challenge is that POSIX shell only has one array to work with. I may be able to inefficiently use more arrays, and I might try just for fun in the future but for now I have something that functions the way I want it to. I would have to use something like Bash if I intend to work more in depth with arrays though.

.sync-0 example

SOURCE_DIRECTORY_PATH="/home"
SOURCE_DIRECTORY_PATH_EXCLUSIONS="--exclude=.sync* --exclude=lost+found --exclude=.cache/*"
DESTINATION_DIRECTORY_PATH="/media/archive/home"

dir-sync script

#!/bin/sh



# VARIABLES
## ENABLE FILE TRANSFER: Change variable `DRY_RUN` to `#DRY_RUN` to enable file tranfers
DRY_RUN="--dry-run" # Disables all file transfers

## OPTIONS (SCRIPT DEFINED/DO NOT TOUCH)
OPTIONS="--archive --acls --one-file-system --xattrs --hard-links --sparse --verbose --human-readable --partial --progress --compress"
OPTIONS_EXTRA="--delete --numeric-ids"



# FUNCTIONS
## SPACER
SPACER() {
    printf "\n\n\n\n\n"
}

## RSYNC ERROR WARNINGS
ERROR_WARNINGS() {
    if [ "$RSYNC_STATUS" -eq 0 ]; then

        # SUCCESSFUL
        printf "\nSync successful"
        printf "\nExit status(0): %s\n" "$RSYNC_STATUS"

    else
        # ERRORS OCCURED
        printf "\nSome error occurred"
        printf "\nExit status(0): %s\n" "$RSYNC_STATUS"
    fi
}

## COPY INPUT STRING ARRAY
COPY_ARRAY()
{
    for i do
        printf %s\\n "$i" | sed "s/'/'\\\\''/g;1s/^/'/;\$s/\$/' \\\\/"
    done
    printf " "
}

## CONFIRMATION (YES/NO)
CONFIRM_YESNO() {
    while true; do
        prompt="${1}"
        printf "%s (Yes/No): " "${prompt}" >&2 # FUNCTION CALL REQUIRES TEXT PROMPT ARGUMENT
        read -r reply
        case $reply in
            [Yy]* ) return 0;; # YES
            [Nn]* ) return 1;; # NO
            * ) printf "Options: y / n\n";;
        esac
    done
}

## DRY RUN
DRY_RUN() {
    # ARRAY SETUP
    eval "set -- ${1}" # ${ARRAY}

    for i do
        # SET CURRENT VARIABLE FROM ARRAY
        current_array="${1}"

        # SET VARIABLES FROM SOURCE FILE
        . "${current_array}"
        source_dir="${SOURCE_DIRECTORY_PATH}"
        exclusions="${SOURCE_DIRECTORY_PATH_EXCLUSIONS}"
        destination_dir="${DESTINATION_DIRECTORY_PATH}"

        # RUN RSYNC DRY RUN COMMAND
        SPACER
        printf "\nStarting %s dry run\n" "${source_dir}"
        rsync --dry-run ${OPTIONS} ${OPTIONS_EXTRA}  ${exclusions} "${source_dir}"/ "${destination_dir}"/
        RSYNC_STATUS=$?
        ERROR_WARNINGS

        # SHIFT TO NEXT VARIABLE IN ARRAY
        shift 1
    done
}

## TRANSFER
TRANSFER() {
    # ARRAY SETUP
    eval "set -- ${1}" # ${ARRAY}

    for i do
        # SET CURRENT VARIABLE FROM ARRAY
        current_array="${1}"

        # SET VARIABLES FROM SOURCE FILE
        . "${current_array}"
        source_dir="${SOURCE_DIRECTORY_PATH}"
        exclusions="${SOURCE_DIRECTORY_PATH_EXCLUSIONS}"
        destination_dir="${DESTINATION_DIRECTORY_PATH}"

        # RUN RSYNC TRANSFER COMMAND
        SPACER
        printf "\nContinuing %s transfer\n" "${source_dir}"
        rsync ${DRY_RUN} ${OPTIONS} ${OPTIONS_EXTRA} ${exclusions} "${source_dir}"/ "${destination_dir}"/
        RSYNC_STATUS=$?
        ERROR_WARNINGS

        # SHIFT TO NEXT VARIABLE IN ARRAY
        shift 1
    done
}



##### START

# CHECK FOR ROOT
if ! [ "$(id -u)" = 0 ]; then

    # EXIT WITH NO ACTIONS TAKEN
    printf "\nRoot access required\n\n"
    return

else
    # ARRAY SETUP
    ARRAY=$(COPY_ARRAY "$@")

    printf "\nStarting transfer process..."

    # SOURCE TO DESTINATION DRY RUN
    DRY_RUN "${ARRAY}"

    # CONFIRM SOURCE TO DESTINATION TRANSFER
    SPACER
    if CONFIRM_YESNO "Proceed with transfer(s)?"; then

        # CONTINUE SOURCE TO DESTINATION TRANSFER & EXIT
        TRANSFER "${ARRAY}"

        printf "\nBackup(s) completed\n\n"
        return

    else
        # SKIP SOURCE TO DESTINATION TRANSFER & EXIT
        printf "\nBackup(s) skipped\n\n"
        return
    fi
fi

##### FINISH

I now have a bunch of other ideas on how I want to improve the script now. Adding ssh/network options, a check to make sure there are no empty command arguments, an option to create a blank template .sync- file in a chosen directory and a reverse direction transfer option are a few things I can think of off the top off my head. Hopefully this array was the most difficult part to learn and understand.

this post was submitted on 26 May 2025

14 points (100.0% liked)

Shell Scripting

1482 readers

2 users here now

From Ash, Bash and Csh to Xonsh, Ysh and Zsh; all shell languages are welcome here!

Rules:

Follow Lemmy rules!
Posts must relate to shell scripting. (See bottom of sidebar for more information.)
Only make helpful replies to questions. This is not the place for low effort joke answers.
No discussion about piracy or hacking.
If you find a solution to your problem by other means, please take your time to write down the steps you used to solve your problem in the original post. You can potentially help others having the same problem!
These rules will change as the community grows.

Keep posts about shell scripting! Here are some guidelines to help:

Allowed: Release Announcement of a command-line program designed for scripted use (e.g. bash, bats, awk, jq, coreutils/moreutils)
Allowed: Tutorials on using shell languages or supplementary tools designed for scripted use
Allowed: Code review/help requests for shell languages or supplementary tools designed for scripted use
NOT Allowed: Announcement of a CLI or TUI program that is not designed for scripted use (Yes, your fancy Zsh plugin which pretty-prints the date and time using only builtins is very cool, but unless you actually want to discuss the code itself, please check out !commandline instead!)
NOT Allowed: Domain-specific tutorials that do not involve shell scripting as a core component (e.g. docker-compose, ansible, nix). If you really love one of these, I'm sure there's a community out there ready to talk about them!
NOT Allowed: Code review requests for non-shell programming languages and configuration languages (e.g. Python, Yaml)

In general, if your submission text is primarily shell code, then it is welcome here!

founded 2 years ago

MODERATORS

[email protected]