[Users] Unknown quota format: vfsv1
tranxene50
tranxene50 at openvz-diff-backups.fr
Tue Sep 22 16:37:56 MSK 2020
Understood.
Can you try using the file "ovzdb.inc" present in attachments ?
Le 22/09/2020 à 15:30, mailinglist at tikklik.nl a écrit :
>
> Oeps my mistake, wrong FIXME line
>
> But sil…
>
> openvz-diff-backups update all auto
>
> /usr/local/sbin/openvz-diff-backups_stable/lib/ovzdb.inc: line 213:
> CMD_BASH: readonly variable
>
> /usr/local/sbin/openvz-diff-backups_stable/lib/ovzdb.inc: line 323:
> BATCH_TASK: unbound variable
>
> “# ---
>
> readonly CMD_BASH="bash-static"
>
> export CMD_BASH
>
> # FIXME (low): hardcoded (CMD_NOCACHE - never called directly)
>
> readonly CMD_NOCACHE="nocache"
>
> export CMD_NOCACHE
>
> “
>
> *Van:* users-bounces at openvz.org <users-bounces at openvz.org> *Namens
> *tranxene50
> *Verzonden:* dinsdag 22 september 2020 15:19
> *Aan:* users at openvz.org
> *Onderwerp:* Re: [Users] Unknown quota format: vfsv1
>
> You have inserted the code at line 138.
>
> please delete it and insert it line 213:
>
> => readonly CMD_BASH="bash-static"
> => export CMD_BASH
>
> # FIXME (low): hardcoded (CMD_NOCACHE - never called directly)
> readonly CMD_NOCACHE="$CMD_NOCACHE"
>
> return 0
>
> To see lines with vi, press ":" and type "set number".
>
> Le 22/09/2020 à 15:06, mailinglist at tikklik.nl
> <mailto:mailinglist at tikklik.nl> a écrit :
>
> Sorry still not working,
> but the quota format error is comming from this vps, it uses 2nd
> level disk quota and is 32bits
> so this VPS will need to migrate the content to a new one.
>
> # --- 2020-09-22 14:59:53 ---> update all auto
>
> ### OVZDB update ### - initiated - 2020-09-22 14:59:57 - update
> all auto
>
> Update - ctid: 007 - initiated - 2020-09-22 14:59:57
>
> /usr/local/sbin/openvz-diff-backups_stable/lib/ovzdb.inc: line
> 138: CMD_BASH: readonly variable
>
> [2020-09-22 14:59:57] [ all] *Error* - openvz-diff-backups
> update 007 auto
>
> *Failure: 0s* - 1 ctid - 2020-09-22 14:59:57 - update all auto
>
> [2020-09-22 14:59:57] [ all] *Error* - ### OVZDB exit: 4 ###
>
> *Van:* users-bounces at openvz.org <mailto:users-bounces at openvz.org>
> <users-bounces at openvz.org> <mailto:users-bounces at openvz.org>
> *Namens *tranxene50
> *Verzonden:* maandag 21 september 2020 14:59
> *Aan:* users at openvz.org <mailto:users at openvz.org>
> *Onderwerp:* Re: [Users] Unknown quota format: vfsv1
>
> Hello Steffan.
>
> Could you try this ?
>
> *** 1) install latest release
>
> # OVZDB_RELEASE="v1.0.1.12-stable"
>
> # cd ////usr/local/sbin////
>
> # wget
> "https://download.openvz-diff-backups.fr/releases/openvz-diff-backups_${OVZDB_RELEASE}.tar.gz"
> <https://download.openvz-diff-backups.fr/releases/openvz-diff-backups_$%7bOVZDB_RELEASE%7d.tar.gz>
>
> # tar xvzf openvz-diff-backups_${OVZDB_RELEASE}.tar.gz
>
> # mv openvz-diff-backups_${OVZDB_RELEASE} openvz-diff-backups_stable
>
> # ln -s openvz-diff-backups_stable/openvz-diff-backups
> openvz-diff-backups
>
> # rm openvz-diff-backups_${OVZDB_RELEASE}.tar.gz
>
> *** 2) add Bash-static
>
> # cd /usr/local/sbin/openvz-diff-backups_stable/
>
> # sed -i "s/\#\!\/bin\/bash /\#\!\/bin\/bash-static /"
> openvz-diff-backups lib/*
>
> # vi lib/ovzdb.inc
>
> Line 213, just above "# FIXME (low): hardcoded (CMD_NOCACHE -
> never called directly)", insert:
>
> readonly CMD_BASH="bash-static"
> export CMD_BASH
>
> *** 3) check
>
> # openvz-diff-backups update all auto
>
> => *Notice* - your release is up to date. Yay!
>
> Normally, backup task should work. :-)
>
> Le 21/09/2020 à 09:44, mailinglist at tikklik.nl
> <mailto:mailinglist at tikklik.nl> a écrit :
>
> Hello Tranxene50,
>
> I cant replase the /bin/bash, so i put it in /bin/bash-static.
>
> Then i needed to replace all files from #!/bin/bash
>
> To #!/bin/bash-static
>
> But that didnt help
>
> # --- 2020-09-21 09:43:05 ---> backup 25
>
> ionice: ignoring given class data for idle class
>
> ionice: ignoring given class data for idle class
>
> ionice: ignoring given class data for idle class
>
> /usr/local/sbin/openvz-diff-backups_v1.0.1.9-stable/lib/ovzdb.inc:
> line 807: declare: -g: invalid option
>
> declare: usage: declare [-aAfFilrtux] [-p] [name[=value] ...]
>
> [2020-09-21 09:43:06] [ 25] *Error* -
> openvz-diff-backups backup 25
>
> *Failure: 0s* - 1 ctid - 2020-09-21 09:43:06 - backup 25
>
> [2020-09-21 09:43:06] [ 25] *Error* - ### OVZDB exit: 4 ###
>
> *Van:* users-bounces at openvz.org
> <mailto:users-bounces at openvz.org> <users-bounces at openvz.org>
> <mailto:users-bounces at openvz.org> *Namens *tranxene50
> *Verzonden:* donderdag 17 september 2020 18:08
> *Aan:* users at openvz.org <mailto:users at openvz.org>
> *Onderwerp:* Re: [Users] Unknown quota format: vfsv1
>
> Hello Steffan.
>
> Unfortunately, I cannot provide a patch: it would require too
> much modifications.
>
> But, after checking with a CentOs 6 container having the same
> bash version (4.1.2), you can use "bash-static" from Debian
> packages.
>
> https://packages.debian.org/jessie/bash-static
>
> The only file needed is "bash-static".
>
> On CentOS, save your "/bin/bash" and put "bash-static" at the
> same place.
>
> openvz-diff-backups should now works.
>
> Have a good day!
>
> Le 17/09/2020 à 15:48, mailinglist at tikklik.nl
> <mailto:mailinglist at tikklik.nl> a écrit :
>
> Hm on openvz6 bash is to old so cant use the script
>
> lib/ovzdb.inc: line 809: declare: -g: invalid option
>
> declare: usage: declare [-aAfFilrtux] [-p] [name[=value] ...]
>
> Installed Packages
>
> bash.x86_64
> 4.1.2-48.el6
>
> *Van:* users-bounces at openvz.org
> <mailto:users-bounces at openvz.org>
> <users-bounces at openvz.org>
> <mailto:users-bounces at openvz.org> *Namens
> *mailinglist at tikklik.nl <mailto:mailinglist at tikklik.nl>
> *Verzonden:* donderdag 17 september 2020 15:19
> *Aan:* 'OpenVZ users' <users at openvz.org>
> <mailto:users at openvz.org>
> *Onderwerp:* Re: [Users] Unknown quota format: vfsv1
>
> You mean making a backup on the openvz6 and then import it
> on openvz7?
>
> I didnt see that in the documentations but i can give it a try
>
> But i find it a strange problem and i dont think the vfsv1
> is the problem
>
> If i look at the mountpoints the other containers also
> have jqfmt=vfsv1
>
> *Van:* users-bounces at openvz.org
> <mailto:users-bounces at openvz.org>
> <users-bounces at openvz.org
> <mailto:users-bounces at openvz.org>> *Namens *Paulo Coghi -
> Coghi IT
> *Verzonden:* donderdag 17 september 2020 14:48
> *Aan:* OpenVZ users <users at openvz.org
> <mailto:users at openvz.org>>
> *Onderwerp:* Re: [Users] Unknown quota format: vfsv1
>
> Hello, Steffan
>
> Can you try openvz-diff-backups and provide your feedback,
> using the updated instructions sent here on the mailing list?
>
> On Thu, Sep 17, 2020 at 8:09 AM <mailinglist at tikklik.nl
> <mailto:mailinglist at tikklik.nl>> wrote:
>
> Trying to make a node empty so i can restore it with
> openvz7
> But this server has a old container of a client
> ovztransfer.sh failes with the message:
> Running quotacheck ...
> quotacheck: Unknown quota format: vfsv1
> Supported formats are:
> vfsold - original quota format
> vfsv0 - new quota format
> rpc - use RPC calls
> xfs - XFS quota format
>
> is there a way i can fix this so this container can
> run under openvz7?
>
> Thanx
>
> Steffan
>
>
> _______________________________________________
> Users mailing list
> Users at openvz.org <mailto:Users at openvz.org>
> https://lists.openvz.org/mailman/listinfo/users
>
>
>
>
>
> _______________________________________________
>
> Users mailing list
>
> Users at openvz.org <mailto:Users at openvz.org>
>
> https://lists.openvz.org/mailman/listinfo/users
>
> --
>
> tranxene50
>
> tranxene50 at openvz-diff-backups.fr <mailto:tranxene50 at openvz-diff-backups.fr>
>
>
>
>
> _______________________________________________
>
> Users mailing list
>
> Users at openvz.org <mailto:Users at openvz.org>
>
> https://lists.openvz.org/mailman/listinfo/users
>
> --
>
> tranxene50
>
> tranxene50 at openvz-diff-backups.fr <mailto:tranxene50 at openvz-diff-backups.fr>
>
>
>
> _______________________________________________
>
> Users mailing list
>
> Users at openvz.org <mailto:Users at openvz.org>
>
> https://lists.openvz.org/mailman/listinfo/users
>
> --
> tranxene50
> tranxene50 at openvz-diff-backups.fr <mailto:tranxene50 at openvz-diff-backups.fr>
>
> _______________________________________________
> Users mailing list
> Users at openvz.org
> https://lists.openvz.org/mailman/listinfo/users
--
tranxene50
tranxene50 at openvz-diff-backups.fr
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openvz.org/pipermail/users/attachments/20200922/56c27396/attachment-0001.html>
-------------- next part --------------
#!/bin/bash-static -efu
# openvz-diff-backups 1.0.1.12 (stable) (2020-06-06)
#
# Please do not edit this file, it will be overwritten when upgrading to a new release
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
# author: tranxene50
# contact: tranxene50 at openvz-diff-backups.fr
# website: www.openvz-diff-backups.fr
#
# Note: all ideas are welcome, I will answer every question as best as I can
# ---
# Note: ALL (bash: force shell buffering)
{
# --- Caller
# Note: ALL (include: bash/dash/ash/sh => single brackets)
if [ -z "${CALLER_TASK:-}" ]; then
echo "What the heck are you doing here? Please, use \"openvz-diff-backups\" command."
echo
echo "Press Control-C now or your session [might|may|could|will] be terminated in 10 seconds (sleep 10 && exit 1)"
sleep 10
exit 1
fi
# --- Init
readonly CALLER_DATE="$CALLER_DATE"
readonly CALLER_TASK="$CALLER_TASK"
readonly CALLER_ARGS="$CALLER_ARGS"
if [[ "${0##*/}" == "$CALLER_TASK" ]]; then
if [[ -t "0" ]]; then
readonly OPTION_TTY="-t"
else
readonly OPTION_TTY=""
fi
readonly EMPTY_STRING=" "
export OPTION_TTY
export EMPTY_STRING
export OPTION_LOGLEVEL="7"
# ---
CONFIG_HOST="$(hostname -f || echo "unknown")"
CONFIG_TASK="${0##*/}"
CONFIG_PATH="$(readlink -e -n -- "$0" || true)"
CONFIG_PATH="${CONFIG_PATH%/*}"
CONFIG_DATE="$(date "+%F %T")"
LOG_HOST="localhost"
LOG_ADDR=""
export CONFIG_HOST
# export CONFIG_TASK
export CONFIG_PATH
# export CONFIG_DATE
# export LOG_HOST
else
readonly OPTION_TTY="$OPTION_TTY"
readonly EMPTY_STRING="$EMPTY_STRING"
CONFIG_TASK="${0##*/}"
CONFIG_TASK="${CONFIG_TASK#ovzdb_}"
CONFIG_TASK="${CONFIG_TASK%.inc}"
CONFIG_DATE="$($CMD_DATE "+%F %T")"
# LOG_HOST="$LOG_HOST"
LOG_ADDR=""
fi
# ---
function check_commands {
# Note: ALL (check_commands: PACKAGE_COREUTILS => essential tools)
local PACKAGE_COREUTILS="cat chmod chown comm cp cut date df du echo head ln ls md5sum mkdir mv nice printf rm rmdir sleep sort stat readlink sha256sum tail tee touch tr wc"
# Note: ALL (check_commands: PACKAGE_[BASH|BC|BZIP2|DEBIANUTILS|DNSUTILS|FINDUTILS|GREP|GAWK|HOSTNAME|MOUNT|OPENSS_CLIENT|PROPCPS|RSYNC|SED|TAR|UTIL_LINUX|WGET] => required tools)
local PACKAGE_BASH="bash-static"
local PACKAGE_BC="bc"
local PACKAGE_BZIP2="bzip2"
local PACKAGE_DEBIANUTILS="which"
local PACKAGE_DNSUTILS="dig"
local PACKAGE_FINDUTILS="find"
local PACKAGE_GREP="grep"
local PACKAGE_GAWK="awk"
local PACKAGE_HOSTNAME="hostname"
local PACKAGE_MOUNT="mount umount"
local PACKAGE_OPENSSH_CLIENT="ssh"
local PACKAGE_PROCPS="kill pgrep"
local PACKAGE_RSYNC="rsync"
local PACKAGE_SED="sed"
local PACKAGE_TAR="tar"
local PACKAGE_UTIL_LINUX="ionice"
local PACKAGE_WGET="wget"
# Note: ALL (check_commands: PACKAGE_PBZIP2 => recommended tools)
local PACKAGE_PBZIP2="pbzip2"
# Note: ALL (check_commands: PACKAGE_[LVM2|PLOOP|PROXMOX|VIRTUOZZO|VZCTL] => optional tools)
local PACKAGE_LVM2="lvcreate lvremove lvs lvscan"
local PACKAGE_PLOOP="ploop"
local PACKAGE_PROXMOX="pvectl pvesm pveversion"
local PACKAGE_VIRTUOZZO="prlctl vzpkg"
local PACKAGE_VZCTL="vzctl vzlist vzpid vztmpl-dl"
# --- Cache
if [[ "${0##*/}" != "$CALLER_TASK" ]]; then
local COMMAND_LIST="$PACKAGE_COREUTILS $PACKAGE_BASH $PACKAGE_BC $PACKAGE_BZIP2 $PACKAGE_DEBIANUTILS $PACKAGE_DNSUTILS $PACKAGE_FINDUTILS $PACKAGE_GREP $PACKAGE_GAWK $PACKAGE_HOSTNAME $PACKAGE_MOUNT $PACKAGE_OPENSSH_CLIENT $PACKAGE_PROCPS $PACKAGE_RSYNC $PACKAGE_SED $PACKAGE_TAR $PACKAGE_UTIL_LINUX $PACKAGE_WGET $PACKAGE_PBZIP2 $PACKAGE_LVM2 $PACKAGE_PLOOP $PACKAGE_PROXMOX $PACKAGE_VIRTUOZZO $PACKAGE_VZCTL"
local COMMAND_LIST="${COMMAND_LIST^^}"
local COMMAND_LIST="${COMMAND_LIST//-/_}"
local COMMAND_LIST="CMD_${COMMAND_LIST// / CMD_}"
local COMMAND_LINK=""
for COMMAND_LINK in $COMMAND_LIST; do
readonly $COMMAND_LINK="${!COMMAND_LINK}"
done
# FIXME (low): hardcoded (CMD_NOCACHE - never called directly)
readonly CMD_NOCACHE="$CMD_NOCACHE"
return 0
fi
# --- Check commands (fetch)
local TEMP_ERRORS="0"
local PACKAGE_LIST="PACKAGE_COREUTILS PACKAGE_BASH PACKAGE_BC PACKAGE_BZIP2 PACKAGE_DNSUTILS PACKAGE_DEBIANUTILS PACKAGE_FINDUTILS PACKAGE_GREP PACKAGE_GAWK PACKAGE_HOSTNAME PACKAGE_MOUNT PACKAGE_OPENSSH_CLIENT PACKAGE_PROCPS PACKAGE_RSYNC PACKAGE_SED PACKAGE_TAR PACKAGE_UTIL_LINUX PACKAGE_WGET PACKAGE_PBZIP2 PACKAGE_LVM2 PACKAGE_PLOOP PACKAGE_PROXMOX PACKAGE_VIRTUOZZO PACKAGE_VZCTL"
local PACKAGE_NAME=""
for PACKAGE_NAME in $PACKAGE_LIST; do
local COMMAND_NAME=""
for COMMAND_NAME in ${!PACKAGE_NAME}; do
local COMMAND_LINK="${COMMAND_NAME^^}"
local COMMAND_LINK="CMD_${COMMAND_LINK//-/_}"
# ---
local COMMAND_FUNC="$(type -P "$COMMAND_NAME")"
if [[ -z "$COMMAND_FUNC" ]] || [[ ! -x "$COMMAND_FUNC" ]]; then
readonly $COMMAND_LINK=""
export $COMMAND_LINK
# ---
if [[ "$COMMAND_NAME" == "pbzip2" ]] ||
[[ "$COMMAND_NAME" == "lvcreate" ]] ||
[[ "$COMMAND_NAME" == "lvremove" ]] ||
[[ "$COMMAND_NAME" == "lvs" ]] ||
[[ "$COMMAND_NAME" == "lvscan" ]] ||
[[ "$COMMAND_NAME" == "ploop" ]] ||
[[ "$COMMAND_NAME" == "pvectl" ]] ||
[[ "$COMMAND_NAME" == "pvesm" ]] ||
[[ "$COMMAND_NAME" == "pveversion" ]] ||
[[ "$COMMAND_NAME" == "prlctl" ]] ||
[[ "$COMMAND_NAME" == "vzpkg" ]] ||
[[ "$COMMAND_NAME" == "vzctl" ]] ||
[[ "$COMMAND_NAME" == "vzlist" ]] ||
[[ "$COMMAND_NAME" == "vzpid" ]] ||
[[ "$COMMAND_NAME" == "vztmpl-dl" ]]; then
continue
fi
local TEMP_ERRORS="$(( TEMP_ERRORS + 1 ))"
# ---
local PACKAGE_LINK="${PACKAGE_NAME#PACKAGE_}"
local PACKAGE_LINK="${PACKAGE_LINK,,}"
local PACKAGE_LINK="${PACKAGE_LINK//_/-}"
echo "[$(date "+%F %T")] *Error* - \"$COMMAND_NAME\" command not found - please install it (\"apt-get install $PACKAGE_LINK\")"
continue
fi
# ---
readonly $COMMAND_LINK="$COMMAND_NAME"
export $COMMAND_LINK
# ---
# Note: ALL (check_commands: debug => display commands in use)
# set +o errexit; set +o noglob; echo -e "\n# --- $COMMAND_LINK\n"; grep -c -r "$COMMAND_LINK" * | column -s ":" -t | grep -v " 0$"; set -o noglob; set -o errexit
done
done
# ---
readonly CMD_BASH="bash-static"
export CMD_BASH
# FIXME (low): hardcoded (CMD_NOCACHE - never called directly)
readonly CMD_NOCACHE="nocache"
export CMD_NOCACHE
# --- Check /dev/fd ($CMD_COMM)
if [[ ! -h "/dev/fd" ]]; then
$CMD_ECHO "[$(date "+%F %T")] *Critical* - nonexistent dev entry (path: \"/dev/fd\") - please create it (\"ln -s /proc/self/fd /dev/fd\")"
local TEMP_ERRORS="$(( TEMP_ERRORS + 1 ))"
fi
# ---
if [[ "$TEMP_ERRORS" -gt 0 ]]; then
$CMD_ECHO -e "\n$TEMP_ERRORS missing tool(s) - aborting\n"
trap - ERR EXIT SIGHUP SIGINT SIGPIPE SIGQUIT SIGTERM SIGUSR1
exit 1
fi
}
# ---
function log_task {
local LOG_CODE="$1"
local LOG_MESG="$2"
# --- Check depth
# if [[ "${#FUNCNAME[@]}" -gt 20 ]]; then
# trap - ERR EXIT SIGHUP SIGINT SIGPIPE SIGQUIT SIGTERM SIGUSR1
# $CMD_ECHO "[$($CMD_DATE "+%F %T")] *Alert* - infinite loop detected - please send a bug report"
# exit 2
# fi
# --- Check args
# if [[ "$LOG_CODE" == *[!0-9]* ]] || [[ "$LOG_CODE" -gt 11 ]] || ( [[ "$LOG_CODE" -gt 0 ]] && [[ -z "$LOG_MESG" ]] ); then
# exit_critical "Invalid log_task call (code: \"$LOG_CODE\" - mesg: \"$LOG_MESG\") - please send a bug report"
# return
# fi
# --- Log skip
if [[ "$LOG_ADDR" == "$LOG_HOST" ]]; then
if [[ "$LOG_CODE" -gt 9 ]] && [[ "$LOG_CODE" -gt "$OPTION_LOGLEVEL" ]]; then
return 0
fi
if [[ "$LOG_HOST" == "localhost" ]] && [[ "$LOG_CODE" -ge 7 ]] && [[ "$OPTION_LOGLEVEL" -le 7 ]]; then
return 0
fi
fi
# --- Log date
local LOG_DATE="$($CMD_DATE "+%F %T")"
# --- Log ctid
if [[ -n "${BACKUP_CTID:-}" ]]; then
local LOG_CTID="${EMPTY_STRING:0:$(( 6 - ${#BACKUP_CTID} ))}${BACKUP_CTID:0:6}"
elif [[ -n "${BATCH_CTID:-}" ]]; then
local LOG_CTID="${EMPTY_STRING:0:$(( 6 - ${#BATCH_CTID} ))}${BATCH_CTID:0:6}"
else
local LOG_CTID=">init<"
fi
# --- Log file
if [[ "$LOG_ADDR" != "$LOG_HOST" ]]; then
# Note: mutable
LOG_ADDR="$LOG_HOST"
# --- Set paths (local)
local LOG_PATH="$CONFIG_PATH/var/log"
if [[ ! -d "$LOG_PATH/" ]]; then
$CMD_MKDIR ${ARGS_CPMKMVRM:-} "$LOG_PATH/" && \
$CMD_CHOWN ${ARGS_CHOWNMOD:-} "root:root" "$LOG_PATH/" && \
$CMD_CHMOD ${ARGS_CHOWNMOD:-} "0750" "$LOG_PATH/"
fi
# ---
if [[ "$LOG_HOST" == "localhost" ]]; then
# Note: mutable
LOG_FILE="$LOG_PATH/ovzdb.log"
else
# Note: mutable
LOG_FILE="$LOG_PATH/ovzdb_${BATCH_TASK}_${LOG_HOST}.log"
fi
# --- Create log file
if [[ ! -f "$LOG_FILE" ]]; then
# Note: ALL (log_task: do not add "2>&1 | $CMD_TEE -a $LOG_FILE" => LOG_FILE inside LOG_FILE)
# Note: ALL (log_task: contention + race condition => touch file)
$CMD_TOUCH "$LOG_FILE" && \
$CMD_CHOWN ${ARGS_CHOWNMOD:-} "root:root" "$LOG_FILE" && \
$CMD_CHMOD ${ARGS_CHOWNMOD:-} "0640" "$LOG_FILE"
else
local LOG_SIZE="$($CMD_DU -k "$LOG_FILE" 2>> "$LOG_FILE" | $CMD_CUT -f "1" -s)"
fi
# --- Separator
$CMD_ECHO -e "\n# --- $CALLER_DATE ---> $BATCH_TASK ${BACKUP_CTID:-$BATCH_CTID} $BATCH_ARGS" >> "$LOG_FILE"
if [[ "$LOG_HOST" != "localhost" ]]; then
$CMD_ECHO "" >> "$LOG_FILE"
fi
# --- Archive log file
# FIXME (low): hardcoded (LOG_SIZE > 16 MB - archive & compress)
if [[ "${LOG_SIZE:-0}" -gt 16384 ]]; then
$CMD_ECHO "[$LOG_DATE] [$LOG_CTID] Info2 | Maximum log size exceeded (file: \"${LOG_FILE##*/}\" - size: \"$LOG_SIZE\" kB) - archiving and compressing log file" >> "$LOG_FILE"
# Note: ALL (log_task: contention + race condition => random delay)
$CMD_SLEEP "$(( RANDOM % 5 )).$(( RANDOM % 10 ))"
# --- Create archive directory
local LOG_ARCH="$CONFIG_PATH/var/log/archives"
if [[ ! -d "$LOG_ARCH/" ]]; then
$CMD_MKDIR ${ARGS_CPMKMVRM:-} "$LOG_ARCH/" 2>&1 | $CMD_TEE -a "$LOG_FILE" && \
$CMD_CHOWN ${ARGS_CHOWNMOD:-} "root:root" "$LOG_ARCH/" 2>&1 | $CMD_TEE -a "$LOG_FILE" && \
$CMD_CHMOD ${ARGS_CHOWNMOD:-} "0750" "$LOG_ARCH/" 2>&1 | $CMD_TEE -a "$LOG_FILE"
fi
# --- Create archive log
if [[ "$LOG_HOST" == "localhost" ]]; then
local LOG_BZIP="$LOG_ARCH/ovzdb_$($CMD_DATE "+%Y-%m-%d_%H-%M-%S").log.bz2"
else
local LOG_BZIP="$LOG_ARCH/ovzdb_${BATCH_TASK}_${LOG_HOST}_$($CMD_DATE "+%Y-%m-%d_%H-%M-%S").log.bz2"
fi
# ---
if [[ ! -f "$LOG_BZIP" ]]; then
# Note: ALL (log_task: contention + race condition => touch file)
$CMD_TOUCH "$LOG_BZIP" 2>&1 | $CMD_TEE -a "$LOG_FILE"
# --- Compress
# Note: ALL (log_task: undefined CMD_MAGIC_BZIP2)
$CMD_IONICE -c 3 $CMD_NICE -n 19 \
${CMD_PBZIP2:-$CMD_BZIP2} -9 -c -z ${ARGS_BZIP2:-} "$LOG_FILE" 2>> "$LOG_FILE" > "$LOG_BZIP" && \
$CMD_CHOWN ${ARGS_CHOWNMOD:-} "root:root" "$LOG_BZIP" 2>&1 | $CMD_TEE -a "$LOG_FILE" && \
$CMD_CHMOD ${ARGS_CHOWNMOD:-} "0640" "$LOG_BZIP" 2>&1 | $CMD_TEE -a "$LOG_FILE"
# --- Truncate
# Note: ALL (log_task: do not add "2>&1 | $CMD_TEE -a $LOG_FILE" => LOG_FILE inside LOG_FILE)
$CMD_CP ${ARGS_CPMKMVRM:-} "/dev/null" "$LOG_FILE" && \
$CMD_CHOWN ${ARGS_CHOWNMOD:-} "root:root" "$LOG_FILE" && \
$CMD_CHMOD ${ARGS_CHOWNMOD:-} "0640" "$LOG_FILE"
fi
fi
# --- Delete & archive log files (5% random)
if [[ "$CONFIG_TASK" == "$CALLER_TASK" ]] && [[ "$(( RANDOM % 100 ))" -lt 5 ]]; then
# Note: ALL (log_task: contention + race condition => random delay)
$CMD_SLEEP "$(( RANDOM % 5 )).$(( RANDOM % 10 ))"
# --- Create archive directory
local LOG_ARCH="$CONFIG_PATH/var/log/archives"
if [[ ! -d "$LOG_ARCH/" ]]; then
$CMD_MKDIR ${ARGS_CPMKMVRM:-} "$LOG_ARCH/" 2>&1 | $CMD_TEE -a "$LOG_FILE" && \
$CMD_CHOWN ${ARGS_CHOWNMOD:-} "root:root" "$LOG_ARCH/" 2>&1 | $CMD_TEE -a "$LOG_FILE" && \
$CMD_CHMOD ${ARGS_CHOWNMOD:-} "0750" "$LOG_ARCH/" 2>&1 | $CMD_TEE -a "$LOG_FILE"
fi
# --- Delete
local TEMP_LIST="$($CMD_FIND "$LOG_ARCH/" -mindepth 1 -maxdepth 1 -type f -name "ovzdb_*.log.bz2" -mtime +180 -printf "%f\n" 2>> "$LOG_FILE" | $CMD_SORT)"
if [[ -n "${TEMP_LIST:-}" ]]; then
$CMD_ECHO "[$LOG_DATE] [$LOG_CTID] Info2 | Deleting log files (3 months old and more)" >> "$LOG_FILE"
local TEMP_NAME=""
for TEMP_NAME in $TEMP_LIST; do
$CMD_ECHO "[$LOG_DATE] [$LOG_CTID] Info2 | - $TEMP_NAME" >> "$LOG_FILE"
$CMD_RM ${ARGS_CPMKMVRM:-} "$LOG_ARCH/$TEMP_NAME" 2>&1 | $CMD_TEE -a "$LOG_FILE"
done
fi
# --- Archive
local TEMP_LIST="$($CMD_FIND "$LOG_PATH/" -mindepth 1 -maxdepth 1 -type f -name "ovzdb_*.log" -mtime +30 -printf "%f\n" 2>> "$LOG_FILE" | $CMD_SORT)"
if [[ -n "${TEMP_LIST:-}" ]]; then
$CMD_ECHO "[$LOG_DATE] [$LOG_CTID] Info2 | Archiving log files (1 month old and more)" >> "$LOG_FILE"
local TEMP_NAME=""
for TEMP_NAME in $TEMP_LIST; do
$CMD_ECHO "[$LOG_DATE] [$LOG_CTID] Info2 | - $TEMP_NAME" >> "$LOG_FILE"
local LOG_STAT="$($CMD_STAT -c "%Y" "$LOG_PATH/$TEMP_NAME")"
local LOG_BZIP="$LOG_ARCH/${TEMP_NAME%.log}_$($CMD_DATE -d "@$LOG_STAT" "+%Y-%m-%d_%H-%M-%S").log.bz2"
# ---
if [[ ! -f "$LOG_BZIP" ]]; then
# Note: ALL (log_task: contention + race condition => touch file)
$CMD_TOUCH "$LOG_BZIP" 2>&1 | $CMD_TEE -a "$LOG_FILE"
# --- Compress
# Note: ALL (log_task: undefined CMD_MAGIC_BZIP2)
$CMD_IONICE -c 3 $CMD_NICE -n 19 \
${CMD_PBZIP2:-$CMD_BZIP2} -9 -c -z ${ARGS_BZIP2:-} "$LOG_PATH/$TEMP_NAME" 2>> "$LOG_FILE" > "$LOG_BZIP" && \
$CMD_CHOWN ${ARGS_CHOWNMOD:-} "root:root" "$LOG_BZIP" 2>&1 | $CMD_TEE -a "$LOG_FILE" && \
$CMD_CHMOD ${ARGS_CHOWNMOD:-} "0640" "$LOG_BZIP" 2>&1 | $CMD_TEE -a "$LOG_FILE"
# --- Delete
$CMD_RM ${ARGS_CPMKMVRM:-} "$LOG_PATH/$TEMP_NAME" 2>&1 | $CMD_TEE -a "$LOG_FILE"
fi
done
fi
fi
fi
# --- Log func
if [[ "$OPTION_LOGLEVEL" -eq 11 ]] && [[ "$LOG_CODE" -gt 0 ]]; then
local FUNCTION="${FUNCNAME[2]:-}"
if [[ -z "$FUNCTION" ]]; then
local FUNCTION="${FUNCNAME[1]:-}"
fi
if [[ -z "$FUNCTION" ]]; then
local FUNCTION="${FUNCNAME[0]:-}"
fi
local LOG_MESG="[${FUNCTION:0:33}]${EMPTY_STRING:0:$(( 33 - ${#FUNCTION} ))} $LOG_MESG"
fi
# --- Log print
if [[ "$LOG_CODE" -le "$OPTION_LOGLEVEL" ]]; then
if [[ "$LOG_MESG" =~ ^dry\-run: ]]; then
$CMD_ECHO "[$LOG_DATE] [$LOG_CTID] Dry-run | ${LOG_MESG#dry-run: }"
elif [[ "$LOG_CODE" -eq 0 ]]; then $CMD_ECHO "${LOG_MESG}"
elif [[ "$LOG_CODE" -eq 1 ]]; then $CMD_ECHO "[$LOG_DATE] [$LOG_CTID] *Emerg* - ${LOG_MESG}"
elif [[ "$LOG_CODE" -eq 2 ]]; then $CMD_ECHO "[$LOG_DATE] [$LOG_CTID] *Alert* - ${LOG_MESG}"
elif [[ "$LOG_CODE" -eq 3 ]]; then $CMD_ECHO "[$LOG_DATE] [$LOG_CTID] *Critical* - ${LOG_MESG}"
elif [[ "$LOG_CODE" -eq 4 ]]; then $CMD_ECHO "[$LOG_DATE] [$LOG_CTID] *Error* - ${LOG_MESG}"
elif [[ "$LOG_CODE" -eq 5 ]]; then $CMD_ECHO "[$LOG_DATE] [$LOG_CTID] *Warning* - ${LOG_MESG}"
elif [[ "$LOG_CODE" -eq 6 ]]; then $CMD_ECHO "[$LOG_DATE] [$LOG_CTID] *Notice* - ${LOG_MESG}"
elif [[ "$LOG_CODE" -eq 7 ]] && ( [[ "$LOG_MESG" =~ ^\# ]] || [[ "$LOG_MESG" =~ ^\< ]] ); then
$CMD_ECHO "[$LOG_DATE] [$LOG_CTID] Info + ${LOG_MESG}"
elif [[ "$LOG_CODE" -eq 7 ]]; then $CMD_ECHO "[$LOG_DATE] [$LOG_CTID] Info | ${LOG_MESG}"
elif [[ "$LOG_CODE" -eq 8 ]]; then $CMD_ECHO "[$LOG_DATE] [$LOG_CTID] Info2 | ${LOG_MESG}"
elif [[ "$LOG_CODE" -eq 9 ]]; then $CMD_ECHO "[$LOG_DATE] [$LOG_CTID] Time | ${LOG_MESG}"
elif [[ "$LOG_CODE" -eq 10 ]]; then $CMD_ECHO "[$LOG_DATE] [$LOG_CTID] Debug | ${LOG_MESG}"
elif [[ "$LOG_CODE" -eq 11 ]]; then $CMD_ECHO "[$LOG_DATE] [$LOG_CTID] Trace | ${LOG_MESG}"
fi
fi
# --- Log write
if [[ "$LOG_CODE" -le "$OPTION_LOGLEVEL" ]] || [[ "$LOG_CODE" -le 9 ]]; then
if [[ "$LOG_MESG" =~ ^dry\-run: ]]; then
$CMD_ECHO "[$LOG_DATE] [$LOG_CTID] Dry-run | ${LOG_MESG#dry-run: }" >> "$LOG_FILE"
elif [[ "$LOG_CODE" -eq 0 ]]; then $CMD_ECHO "${LOG_MESG}" >> "$LOG_FILE"
elif [[ "$LOG_CODE" -eq 1 ]]; then $CMD_ECHO "[$LOG_DATE] [$LOG_CTID] *Emerg* - ${LOG_MESG}" >> "$LOG_FILE"
elif [[ "$LOG_CODE" -eq 2 ]]; then $CMD_ECHO "[$LOG_DATE] [$LOG_CTID] *Alert* - ${LOG_MESG}" >> "$LOG_FILE"
elif [[ "$LOG_CODE" -eq 3 ]]; then $CMD_ECHO "[$LOG_DATE] [$LOG_CTID] *Critical* - ${LOG_MESG}" >> "$LOG_FILE"
elif [[ "$LOG_CODE" -eq 4 ]]; then $CMD_ECHO "[$LOG_DATE] [$LOG_CTID] *Error* - ${LOG_MESG}" >> "$LOG_FILE"
elif [[ "$LOG_CODE" -eq 5 ]]; then $CMD_ECHO "[$LOG_DATE] [$LOG_CTID] *Warning* - ${LOG_MESG}" >> "$LOG_FILE"
elif [[ "$LOG_CODE" -eq 6 ]]; then $CMD_ECHO "[$LOG_DATE] [$LOG_CTID] *Notice* - ${LOG_MESG}" >> "$LOG_FILE"
elif [[ "$LOG_CODE" -eq 7 ]] && ( [[ "$LOG_MESG" =~ ^\# ]] || [[ "$LOG_MESG" =~ ^\< ]] ); then
$CMD_ECHO "[$LOG_DATE] [$LOG_CTID] Info + ${LOG_MESG}" >> "$LOG_FILE"
elif [[ "$LOG_CODE" -eq 7 ]]; then $CMD_ECHO "[$LOG_DATE] [$LOG_CTID] Info | ${LOG_MESG}" >> "$LOG_FILE"
elif [[ "$LOG_CODE" -eq 8 ]]; then $CMD_ECHO "[$LOG_DATE] [$LOG_CTID] Info2 | ${LOG_MESG}" >> "$LOG_FILE"
elif [[ "$LOG_CODE" -eq 9 ]]; then $CMD_ECHO "[$LOG_DATE] [$LOG_CTID] Time | ${LOG_MESG}" >> "$LOG_FILE"
elif [[ "$LOG_CODE" -eq 10 ]]; then $CMD_ECHO "[$LOG_DATE] [$LOG_CTID] Debug | ${LOG_MESG}" >> "$LOG_FILE"
elif [[ "$LOG_CODE" -eq 11 ]]; then $CMD_ECHO "[$LOG_DATE] [$LOG_CTID] Trace | ${LOG_MESG}" >> "$LOG_FILE"
fi
fi
}
# --- Alias
function log_echo { log_task 0 "$1"; }
function log_emerg { log_task 1 "$1"; }
function log_alert { log_task 2 "$1"; }
function log_critical { log_task 3 "$1"; }
function log_error { log_task 4 "$1"; }
function log_warning { log_task 5 "$1"; }
function log_notice { log_task 6 "$1"; }
function log_info { log_task 7 "$1"; }
function log_info2 { log_task 8 "$1"; }
function log_time { log_task 9 "$1"; }
function log_debug { log_task 10 "$1"; }
function log_trace { log_task 11 "$1"; }
# ---
function exit_task {
local EXIT_CODE="$1"
local EXIT_MESG="$2"
# --- Check depth
# if [[ "${#FUNCNAME[@]}" -gt 20 ]]; then
# trap - ERR EXIT SIGHUP SIGINT SIGPIPE SIGQUIT SIGTERM SIGUSR1
# $CMD_ECHO "[$($CMD_DATE "+%F %T")] *Alert* - infinite loop detected - please send a bug report"
# exit 2
# fi
# --- Check args
# if [[ "$EXIT_CODE" == *[!0-9]* ]] || [[ "$EXIT_CODE" -gt 11 ]] || ( [[ "$EXIT_CODE" -gt 0 ]] && [[ -z "$EXIT_MESG" ]] ); then
# exit_critical "Invalid exit_task call (code: \"$EXIT_CODE\" - mesg: \"$EXIT_MESG\") - please send a bug report"
# return
# fi
# ---
if [[ "$EXIT_CODE" -eq 0 ]]; then log_task 7 "# --- Exit status: success"
elif [[ "$EXIT_CODE" -eq 1 ]]; then log_task 1 "$EXIT_MESG"; log_task 7 "# --- Exit status: emerg"
elif [[ "$EXIT_CODE" -eq 2 ]]; then log_task 2 "$EXIT_MESG"; log_task 7 "# --- Exit status: alert"
elif [[ "$EXIT_CODE" -eq 3 ]]; then log_task 3 "$EXIT_MESG"; log_task 7 "# --- Exit status: critical"
elif [[ "$EXIT_CODE" -eq 4 ]]; then log_task 4 "$EXIT_MESG"; log_task 7 "# --- Exit status: error"
elif [[ "$EXIT_CODE" -eq 5 ]]; then log_task 5 "$EXIT_MESG"; log_task 7 "# --- Exit status: warning"
elif [[ "$EXIT_CODE" -eq 6 ]]; then log_task 6 "$EXIT_MESG"; log_task 7 "# --- Exit status: notice"
elif [[ "$EXIT_CODE" -eq 7 ]]; then log_task 7 "$EXIT_MESG"; log_task 7 "# --- Exit status: info"
elif [[ "$EXIT_CODE" -eq 8 ]]; then log_task 8 "$EXIT_MESG"; log_task 7 "# --- Exit status: info2"
elif [[ "$EXIT_CODE" -eq 9 ]]; then log_task 9 "$EXIT_MESG"; log_task 7 "# --- Exit status: time"
elif [[ "$EXIT_CODE" -eq 10 ]]; then log_task 10 "$EXIT_MESG"; log_task 7 "# --- Exit status: debug"
elif [[ "$EXIT_CODE" -eq 10 ]]; then log_task 11 "$EXIT_MESG"; log_task 7 "# --- Exit status: trace"
fi
# --- Check lock
if [[ -n "${EXIT_LOCK:-}" ]]; then
log_notice "Exit already in progress - ignoring (please, do not panic)"
return 0
fi
readonly EXIT_LOCK="$CONFIG_TASK"
# --- Exit tasks
if [[ -n "${EXIT_FUNCTIONS:-}" ]]; then
declare -a TEMP_ARRAY="()"
IFS=" " read -a TEMP_ARRAY <<< "$EXIT_FUNCTIONS"
local TEMP_INDEX="0"
local TEMP_COUNT="${#TEMP_ARRAY[@]}"
while [[ "$TEMP_INDEX" -lt "$TEMP_COUNT" ]]; do
local EXIT_FUNCTION="${TEMP_ARRAY[$TEMP_INDEX]}"
log_notice "Executing exit function (name: \"$EXIT_FUNCTION\") - please, do not interrupt"
task_pause
$EXIT_FUNCTION
local TEMP_INDEX="$(( TEMP_INDEX + 1 ))"
done
fi
# --- Error
if [[ "$EXIT_CODE" -ne 0 ]]; then
if [[ "$CONFIG_TASK" == "backup" ]] || [[ "$CONFIG_TASK" == "delete" ]] || [[ "$CONFIG_TASK" == "destroy" ]] || [[ "$CONFIG_TASK" == "download" ]] || [[ "$CONFIG_TASK" == "upload" ]]; then
local TASK_NAME="$CONFIG_TASK"
fi
# --- Tasks
if [[ -n "${TASK_NAME:-}" ]] && [[ -n "${REMOTE_TASK:-}" ]] && [[ "${BACKUP_MODE:-}" =~ ^(cold|hold|live)_(sync|snap|plop)$ ]]; then
update_task_file "remote" "$TASK_NAME|failure|$BACKUP_MODE|$CONFIG_HOST|unknown|0000-00-00_00-00-00"
fi
if [[ -n "${TASK_NAME:-}" ]] && [[ -n "${MASTER_TASK:-}" ]] && [[ "${BACKUP_MODE:-}" =~ ^(cold|hold|live)_(sync|snap|plop)$ ]]; then
update_task_file "master" "$TASK_NAME|failure|$BACKUP_MODE|$CONFIG_HOST|unknown|0000-00-00_00-00-00"
fi
# --- Unlock
if [[ -n "${REMOTE_LOCK:-}" ]]; then
unlock_storage "remote"
fi
if [[ -n "${MASTER_LOCK:-}" ]]; then
unlock_storage "master"
fi
if [[ -n "${SOURCE_LOCK:-}" ]]; then
unlock_container
fi
# --- Hook "failure"
if [[ -n "${PREVIOUS_HOOK:-}" ]]; then
execute_hook "failure"
fi
fi
# --- Hook "exit"
if [[ -n "${PREVIOUS_HOOK:-}" ]]; then
execute_hook "exit"
fi
# --- Close SSH mux
if [[ "$CONFIG_TASK" == "$CALLER_TASK" ]] && [[ -n "$BATCH_ARGS" ]]; then
if [[ -z "${BATCH_STRG:-}" ]]; then
local BATCH_STRG="${BATCH_ARGS#* }"
local BATCH_STRG="${BATCH_STRG%% *}"
if [[ "$BATCH_STRG" != "master" ]] && [[ "$BATCH_STRG" != "remote" ]]; then
local BATCH_STRG="master"
fi
fi
# ---
if [[ -n "${REMOTE_SSH_HOST:-}" ]] && [[ -n "${REMOTE_SSH_OPTIONS:-}" ]] && [[ "${REMOTE_SSH_OPTIONS,,}" == *"controlpath"* ]]; then
if [[ "$BATCH_TASK" == "download" ]] || [[ "$BATCH_TASK" == "upload" ]] ||
( [[ "$BATCH_TASK" == "delete" ]] && [[ "$BATCH_STRG" == "remote" ]] ) ||
( [[ "$BATCH_TASK" == "destroy" ]] && [[ "$BATCH_STRG" == "remote" ]] ); then
log_info2 "Closing remote SSH mux (task: \"$BATCH_TASK\")"
set +o errexit; trap - ERR
# Note: ALL (exit_task: STDERR prints "Stop listening request sent")
# Note: ALL (exit_task: "stop" remote SSH mux instead of "exit" => ConnectTimeout)
$MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "$REMOTE_SSH_OPTIONS -O stop \"$REMOTE_SSH_HOST\"" 2> "/dev/null" >> "$LOG_FILE"
local ERROR_CODE="$?"
reset_trap
if [[ "$ERROR_CODE" -ne 0 ]]; then
log_notice "SSH error: unable to close remote SSH mux (host: \"$MASTER_SSH_HOST\" -> \"$REMOTE_SSH_HOST\" - error: \"$ERROR_CODE\") - ignoring (please, do not panic)"
fi
fi
fi
# ---
if [[ -n "${MASTER_SSH_HOST:-}" ]] && [[ -n "${MASTER_SSH_OPTIONS:-}" ]] && [[ "${MASTER_SSH_OPTIONS,,}" == *"controlpath"* ]]; then
if [[ "$BATCH_TASK" == "backup" ]] || [[ "$BATCH_TASK" == "delete" ]] || [[ "$BATCH_TASK" == "destroy" ]] || [[ "$BATCH_TASK" == "download" ]] || [[ "$BATCH_TASK" == "upload" ]] || [[ "$BATCH_TASK" == "restore" ]]; then
log_info2 "Closing master SSH mux (task: \"$BATCH_TASK\")"
set +o errexit; trap - ERR
# Note: ALL (exit_task: STDERR prints "Stop listening request sent")
# Note: ALL (exit_task: "stop" master SSH mux instead of "exit" => ConnectTimeout)
$MASTER_SSH_OPTIONS -O stop "$MASTER_SSH_HOST" 2> "/dev/null" >> "$LOG_FILE"
local ERROR_CODE="$?"
reset_trap
if [[ "$ERROR_CODE" -ne 0 ]]; then
log_notice "SSH error: unable to close master SSH mux (host: \"$MASTER_SSH_HOST\" - error: \"$ERROR_CODE\") - ignoring (please, do not panic)"
fi
fi
fi
fi
# ---
task_backtrace
# ---
if [[ "$EXIT_CODE" -eq 0 ]]; then
log_info2 "Task complete (exit code: \"$EXIT_CODE\" - status: \"success\") - exiting now"
else
log_info2 "Task complete (exit code: \"$EXIT_CODE\" - status: \"failure\") - exiting now"
fi
# ---
trap - ERR EXIT SIGHUP SIGINT SIGPIPE SIGQUIT SIGTERM SIGUSR1
exit $EXIT_CODE
}
# --- Alias
function exit_success { exit_task 0 "$1"; return; }
function exit_emerg { exit_task 1 "$1"; return; }
function exit_alert { exit_task 2 "$1"; return; }
function exit_critical { exit_task 3 "$1"; return; }
function exit_error { exit_task 4 "$1"; return; }
function exit_warning { exit_task 5 "$1"; return; }
function exit_notice { exit_task 6 "$1"; return; }
function exit_info { exit_task 7 "$1"; return; }
function exit_info2 { exit_task 8 "$1"; return; }
function exit_time { exit_task 9 "$1"; return; }
function exit_debug { exit_task 10 "$1"; return; }
function exit_trace { exit_task 11 "$1"; return; }
# ---
# FIXME (todo): https://twitter.com/jaybaeta/status/1032825246649221120
function set_trap {
trap "exit_alert \"Unexpected error - error: \\\"$1\\\" - code: \\\"\$?\\\" - stack: \\\"\${BASH_LINENO[*]}\\\" - command: \\\"\$BASH_COMMAND\\\"\"; exit 2;" ERR SIGHUP SIGPIPE SIGQUIT SIGTERM SIGUSR1
trap "exit_critical \"Interrupted with Control-C - error: \\\"$1\\\" - code: \\\"\$?\\\" - stack: \\\"\${BASH_LINENO[*]}\\\" - command: \\\"\$BASH_COMMAND\\\"\"; exit 3;" SIGINT
trap "exit_error \"Unexpected task termination - code: \\\"\$?\\\" - please send a bug report\"; exit 4;" EXIT
set -o errexit; set +o errtrace; set -o noglob; set -o nounset; set -o pipefail; shopt -s failglob
}
# ---
function reset_trap {
trap "exit_alert \"Unexpected error - code: \\\"\$?\\\" - stack: \\\"\${BASH_LINENO[*]}\\\" - command: \\\"\$BASH_COMMAND\\\"\"; exit 2;" ERR SIGHUP SIGPIPE SIGQUIT SIGTERM SIGUSR1
trap "exit_critical \"Interrupted with Control-C - code: \\\"\$?\\\" - stack: \\\"\${BASH_LINENO[*]}\\\" - command: \\\"\$BASH_COMMAND\\\"\"; exit 3;" SIGINT
trap "exit_error \"Unexpected task termination - code: \\\"\$?\\\" - please send a bug report\"; exit 4;" EXIT
set -o errexit; set +o errtrace; set -o noglob; set -o nounset; set -o pipefail; shopt -s failglob
}
# ---
function dry_run {
log_info2 "dry-run: $($CMD_ECHO "${1//$'\n'/ }" | $CMD_TR -s " ")"
if [[ "$OPTION_LOGLEVEL" -gt 9 ]]; then
task_pause 3 force
elif [[ "$OPTION_LOGLEVEL" -gt 7 ]]; then
$CMD_SLEEP 1
fi
}
# ---
function task_backtrace {
if [[ "${OPTION_BACKTRACE:-0}" -eq 1 ]]; then
$CMD_ECHO "--- Backtrace"
local CALL_DEPTH="0"
while caller $CALL_DEPTH; do
local CALL_DEPTH="$(( CALL_DEPTH + 1 ))"
done
$CMD_ECHO "---"
fi
}
# ---
function task_pause {
local PAUSE_DELAY="${1:-3}"
local PAUSE_LEVEL="${OPTION_PAUSE:-0}"
if [[ "${2:-}" == "force" ]]; then
local PAUSE_LEVEL="2"
fi
if [[ "$PAUSE_LEVEL" -eq 0 ]] || [[ -n "${EXIT_LOCK:-}" ]]; then
return 0
fi
# ---
if [[ "$OPTION_LOGLEVEL" -gt 7 ]] || ( [[ "$PAUSE_LEVEL" -eq 1 ]] && [[ "$OPTION_LOGLEVEL" -gt 5 ]] ); then
$CMD_ECHO -n "[$($CMD_DATE "+%F %T")] Pause | "
while [[ "$PAUSE_DELAY" -gt 0 ]]; do
$CMD_ECHO -n "$PAUSE_DELAY... "
$CMD_SLEEP 1
local PAUSE_DELAY="$(( PAUSE_DELAY - 1 ))"
done
$CMD_ECHO ""
else
$CMD_SLEEP $PAUSE_DELAY
fi
}
# ---
function pad_left {
if [[ "${4:-}" == "ro" ]]; then
declare -g -r $2="${EMPTY_STRING:0:$(( $1 - ${#3} ))}${3:0:$1}"
else
declare -g $2="${EMPTY_STRING:0:$(( $1 - ${#3} ))}${3:0:$1}"
fi
}
# ---
function pad_right {
if [[ "${4:-}" == "ro" ]]; then
declare -g -r $2="${3:0:$1}${EMPTY_STRING:0:$(( $1 - ${#3} ))}"
else
declare -g $2="${3:0:$1}${EMPTY_STRING:0:$(( $1 - ${#3} ))}"
fi
}
# ---
function set_speed {
local SPEED_MESG="${1:-}"
local SPEED_SYNC="${2:-}"
local SPEED_MISC="${3:-}"
# Note: mutable
unset SPEED_SOURCE_RSYNC_COMMAND
unset SPEED_MASTER_RSYNC_OPTIONS
unset SPEED_MASTER_RSYNC_COMMAND
unset SPEED_REMOTE_RSYNC_OPTIONS
unset SPEED_REMOTE_RSYNC_COMMAND
# unset SPEED_RESCUE_RSYNC_OPTIONS
# Note: mutable
unset IONICE_CLASS
unset IONICE_LEVEL
unset NICE_LEVEL
unset NICE_SPEED
# ---
if [[ "$SPEED_SYNC" == "sync" ]] && [[ "${BACKUP_MODE:-}" =~ ^(cold|hold|live)_sync$ ]]; then
log_debug "$SPEED_MESG (full speed - highest priority)"
IONICE_CLASS="1"
IONICE_LEVEL="7"
NICE_LEVEL="-20"
NICE_SPEED="0"
elif [[ "$OPTION_TURBO" -gt 2 ]]; then
log_debug "$SPEED_MESG (full speed - higher priority)"
# FIXME (low): hardcoded (Compressing memory dump - pbzip2 is recommended)
if [[ "$SPEED_SYNC" == "nice" ]] && [[ -z "$CMD_PBZIP2" ]] && [[ "$SPEED_MESG" == "Compressing memory dump" ]]; then
log_notice "You should install \"pbzip2\" (Parallel BZIP2) - compression will be much faster"
fi
IONICE_CLASS="2"
IONICE_LEVEL="0"
NICE_LEVEL="-10"
NICE_SPEED="0"
elif [[ "$OPTION_TURBO" -gt 1 ]]; then
log_debug "$SPEED_MESG (full speed - normal priority)"
# FIXME (low): hardcoded (Compressing memory dump - pbzip2 is recommended)
if [[ "$SPEED_SYNC" == "nice" ]] && [[ -z "$CMD_PBZIP2" ]] && [[ "$SPEED_MESG" == "Compressing memory dump" ]]; then
log_notice "You should install \"pbzip2\" (Parallel BZIP2) - compression will be much faster"
fi
IONICE_CLASS="3"
IONICE_LEVEL="0"
NICE_LEVEL="0"
NICE_SPEED="0"
elif [[ "$OPTION_TURBO" -gt 0 ]]; then
log_debug "$SPEED_MESG (full speed - lowest priority)"
IONICE_CLASS="3"
IONICE_LEVEL="0"
NICE_LEVEL="19"
NICE_SPEED="0"
else
if [[ -z "$SPEED_MISC" ]] || [[ "$SPEED_MISC" -eq 0 ]]; then
log_debug "$SPEED_MESG (full speed - lowest priority)"
else
log_debug "$SPEED_MESG (throttled: ~$SPEED_MISC kB/s - lowest priority)"
fi
IONICE_CLASS="3"
IONICE_LEVEL="0"
NICE_LEVEL="19"
NICE_SPEED="$SPEED_MISC"
fi
# ---
if [[ "$SPEED_SYNC" != "nice" ]]; then
# --- Source
if [[ "$IONICE_CLASS" -eq "3" ]]; then
SPEED_SOURCE_RSYNC_COMMAND="$CMD_IONICE -c "$IONICE_CLASS" $CMD_NICE -n "$NICE_LEVEL" $CMD_SOURCE_NOCACHE $CMD_RSYNC --bwlimit=$NICE_SPEED"
else
SPEED_SOURCE_RSYNC_COMMAND="$CMD_IONICE -c "$IONICE_CLASS" -n "$IONICE_LEVEL" $CMD_NICE -n "$NICE_LEVEL" $CMD_SOURCE_NOCACHE $CMD_RSYNC --bwlimit=$NICE_SPEED"
fi
# --- Master
if [[ -n "${MASTER_SSH_HOST:-}" ]]; then
if [[ "$MASTER_IONICE" == "yes" ]] && [[ "$IONICE_CLASS" -eq "3" ]] && [[ "$MASTER_NICE" == "yes" ]]; then
SPEED_MASTER_RSYNC_OPTIONS="$CMD_IONICE -c "$IONICE_CLASS" $CMD_NICE -n "$NICE_LEVEL" $CMD_MASTER_NOCACHE $CMD_RSYNC"
SPEED_MASTER_RSYNC_COMMAND="$CMD_IONICE -c "$IONICE_CLASS" $CMD_NICE -n "$NICE_LEVEL" $CMD_MASTER_NOCACHE $CMD_RSYNC --bwlimit=$NICE_SPEED"
elif [[ "$MASTER_IONICE" == "yes" ]] && [[ "$IONICE_CLASS" -eq "3" ]]; then
SPEED_MASTER_RSYNC_OPTIONS="$CMD_IONICE -c "$IONICE_CLASS" $CMD_MASTER_NOCACHE $CMD_RSYNC"
SPEED_MASTER_RSYNC_COMMAND="$CMD_IONICE -c "$IONICE_CLASS" $CMD_MASTER_NOCACHE $CMD_RSYNC --bwlimit=$NICE_SPEED"
elif [[ "$MASTER_IONICE" == "yes" ]] && [[ "$MASTER_NICE" == "yes" ]]; then
SPEED_MASTER_RSYNC_OPTIONS="$CMD_IONICE -c "$IONICE_CLASS" -n "$IONICE_LEVEL" $CMD_NICE -n "$NICE_LEVEL" $CMD_MASTER_NOCACHE $CMD_RSYNC"
SPEED_MASTER_RSYNC_COMMAND="$CMD_IONICE -c "$IONICE_CLASS" -n "$IONICE_LEVEL" $CMD_NICE -n "$NICE_LEVEL" $CMD_MASTER_NOCACHE $CMD_RSYNC --bwlimit=$NICE_SPEED"
elif [[ "$MASTER_IONICE" == "yes" ]]; then
SPEED_MASTER_RSYNC_OPTIONS="$CMD_IONICE -c "$IONICE_CLASS" -n "$IONICE_LEVEL" $CMD_MASTER_NOCACHE $CMD_RSYNC"
SPEED_MASTER_RSYNC_COMMAND="$CMD_IONICE -c "$IONICE_CLASS" -n "$IONICE_LEVEL" $CMD_MASTER_NOCACHE $CMD_RSYNC --bwlimit=$NICE_SPEED"
elif [[ "$MASTER_NICE" == "yes" ]]; then
SPEED_MASTER_RSYNC_OPTIONS="$CMD_NICE -n "$NICE_LEVEL" $CMD_MASTER_NOCACHE $CMD_RSYNC"
SPEED_MASTER_RSYNC_COMMAND="$CMD_NICE -n "$NICE_LEVEL" $CMD_MASTER_NOCACHE $CMD_RSYNC --bwlimit=$NICE_SPEED"
else
SPEED_MASTER_RSYNC_OPTIONS="$CMD_MASTER_NOCACHE $CMD_RSYNC"
SPEED_MASTER_RSYNC_COMMAND="$CMD_MASTER_NOCACHE $CMD_RSYNC --bwlimit=$NICE_SPEED"
fi
fi
# --- Remote
if [[ -n "${REMOTE_SSH_HOST:-}" ]]; then
if [[ "$REMOTE_IONICE" == "yes" ]] && [[ "$IONICE_CLASS" -eq "3" ]] && [[ "$REMOTE_NICE" == "yes" ]]; then
SPEED_REMOTE_RSYNC_OPTIONS="$CMD_IONICE -c "$IONICE_CLASS" $CMD_NICE -n "$NICE_LEVEL" $CMD_REMOTE_NOCACHE $CMD_RSYNC"
SPEED_REMOTE_RSYNC_COMMAND="$CMD_IONICE -c "$IONICE_CLASS" $CMD_NICE -n "$NICE_LEVEL" $CMD_REMOTE_NOCACHE $CMD_RSYNC --bwlimit=$NICE_SPEED"
elif [[ "$REMOTE_IONICE" == "yes" ]] && [[ "$IONICE_CLASS" -eq "3" ]]; then
SPEED_REMOTE_RSYNC_OPTIONS="$CMD_IONICE -c "$IONICE_CLASS" $CMD_REMOTE_NOCACHE $CMD_RSYNC"
SPEED_REMOTE_RSYNC_COMMAND="$CMD_IONICE -c "$IONICE_CLASS" $CMD_REMOTE_NOCACHE $CMD_RSYNC --bwlimit=$NICE_SPEED"
elif [[ "$REMOTE_IONICE" == "yes" ]] && [[ "$REMOTE_NICE" == "yes" ]]; then
SPEED_REMOTE_RSYNC_OPTIONS="$CMD_IONICE -c "$IONICE_CLASS" -n "$IONICE_LEVEL" $CMD_NICE -n "$NICE_LEVEL" $CMD_REMOTE_NOCACHE $CMD_RSYNC"
SPEED_REMOTE_RSYNC_COMMAND="$CMD_IONICE -c "$IONICE_CLASS" -n "$IONICE_LEVEL" $CMD_NICE -n "$NICE_LEVEL" $CMD_REMOTE_NOCACHE $CMD_RSYNC --bwlimit=$NICE_SPEED"
elif [[ "$REMOTE_IONICE" == "yes" ]]; then
SPEED_REMOTE_RSYNC_OPTIONS="$CMD_IONICE -c "$IONICE_CLASS" -n "$IONICE_LEVEL" $CMD_REMOTE_NOCACHE $CMD_RSYNC"
SPEED_REMOTE_RSYNC_COMMAND="$CMD_IONICE -c "$IONICE_CLASS" -n "$IONICE_LEVEL" $CMD_REMOTE_NOCACHE $CMD_RSYNC --bwlimit=$NICE_SPEED"
elif [[ "$REMOTE_NICE" == "yes" ]]; then
SPEED_REMOTE_RSYNC_OPTIONS="$CMD_NICE -n "$NICE_LEVEL" $CMD_REMOTE_NOCACHE $CMD_RSYNC"
SPEED_REMOTE_RSYNC_COMMAND="$CMD_NICE -n "$NICE_LEVEL" $CMD_REMOTE_NOCACHE $CMD_RSYNC --bwlimit=$NICE_SPEED"
else
SPEED_REMOTE_RSYNC_OPTIONS="$CMD_REMOTE_NOCACHE $CMD_RSYNC"
SPEED_REMOTE_RSYNC_COMMAND="$CMD_REMOTE_NOCACHE $CMD_RSYNC --bwlimit=$NICE_SPEED"
fi
fi
fi
}
# ---
function check_temp_path {
local TEMP_PATH="${1:-undefined}"
local TEMP_STRG="${2:-undefined}"
local TEMP_MODE="${3:-undefined}"
local TEMP_TRAP="${4:-undefined}"
if [[ "$TEMP_PATH" == "undefined" ]] || [[ "$TEMP_PATH" == "/" ]] || [[ "$TEMP_PATH" == "//" ]] || [[ "$TEMP_PATH" == "///" ]] || [[ "${TEMP_PATH:0:1}" != "/" ]] ||
( [[ "$TEMP_STRG" != "source" ]] && [[ "$TEMP_STRG" != "master" ]] && [[ "$TEMP_STRG" != "remote" ]] ) ||
( [[ "$TEMP_MODE" != "exists_soft" ]] && [[ "$TEMP_MODE" != "exists_hard" ]] && [[ "$TEMP_MODE" != "delete_soft" ]] && [[ "$TEMP_MODE" != "delete_hard" ]] ) ||
[[ "$TEMP_TRAP" == "undefined" ]]; then
exit_critical "Invalid check_temp_path call (path: \"$TEMP_PATH\" - strg: \"$TEMP_STRG\" - mode: \"$TEMP_MODE\" - trap: \"$TEMP_TRAP\") - please send a bug report"
return
fi
# Note: PCS6 (check_temp_path: negative offset not available => bash 4.1.2)
if [[ "${TEMP_PATH:$(( ${#TEMP_PATH} - 1 )):1}" == "/" ]]; then
local TEMP_TYPE="path"
else
local TEMP_TYPE="file"
fi
# --- Exists
if [[ "$TEMP_MODE" == "exists_soft" ]] || [[ "$TEMP_MODE" == "exists_hard" ]]; then
set +o errexit; trap - ERR
if [[ "$TEMP_STRG" == "source" ]]; then
if [[ "$TEMP_TYPE" == "path" ]]; then
if [[ -d "$TEMP_PATH" ]]; then true; else (exit 42); fi
elif [[ "$TEMP_TYPE" == "file" ]]; then
if [[ -f "$TEMP_PATH" ]]; then true; else (exit 42); fi
fi
elif [[ "$TEMP_STRG" == "master" ]]; then
if [[ "$TEMP_TYPE" == "path" ]]; then
# Note: ALL (check_temp_path: MASTER_SSH_HOST => single brackets)
$MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "if [ -d \"$TEMP_PATH\" ]; then exit 0; else exit 42; fi" 2>&1 | $CMD_TEE -a "$LOG_FILE"
elif [[ "$TEMP_TYPE" == "file" ]]; then
# Note: ALL (check_temp_path: MASTER_SSH_HOST => single brackets)
$MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "if [ -f \"$TEMP_PATH\" ]; then exit 0; else exit 42; fi" 2>&1 | $CMD_TEE -a "$LOG_FILE"
fi
elif [[ "$TEMP_STRG" == "remote" ]]; then
if [[ "$TEMP_TYPE" == "path" ]]; then
# Note: ALL (check_temp_path: REMOTE_SSH_HOST => single brackets)
$MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "$REMOTE_SSH_OPTIONS \"$REMOTE_SSH_HOST\" \"if [ -d \\\"$TEMP_PATH\\\" ]; then exit 0; else exit 42; fi\"" 2>&1 | $CMD_TEE -a "$LOG_FILE"
elif [[ "$TEMP_TYPE" == "file" ]]; then
# Note: ALL (check_temp_path: REMOTE_SSH_HOST => single brackets)
$MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "$REMOTE_SSH_OPTIONS \"$REMOTE_SSH_HOST\" \"if [ -f \\\"$TEMP_PATH\\\" ]; then exit 0; else exit 42; fi\"" 2>&1 | $CMD_TEE -a "$LOG_FILE"
fi
fi
local ERROR_CODE="$?"
set_trap "$TEMP_TRAP"
# --- Check error code
if [[ "$ERROR_CODE" -ne 0 ]]; then
if [[ "$ERROR_CODE" -eq 255 ]]; then
if [[ "$TEMP_STRG" == "master" ]]; then
exit_critical "SSH error: connection failure (host: \"$MASTER_SSH_HOST\" - error: \"$ERROR_CODE\") - aborting"
return
elif [[ "$TEMP_STRG" == "remote" ]]; then
exit_critical "SSH error: connection failure (host: \"$MASTER_SSH_HOST\" -> \"$REMOTE_SSH_HOST\" - error: \"$ERROR_CODE\") - aborting"
return
fi
fi
# ---
if [[ "$ERROR_CODE" -ne 42 ]]; then
if [[ "$TEMP_STRG" == "source" ]]; then
exit_critical "Unable to check temp file/directory (path: \"$TEMP_PATH\" - error: \"$ERROR_CODE\") - please send a bug report"
return
elif [[ "$TEMP_STRG" == "master" ]]; then
exit_critical "Unable to check temp file/directory (host: \"$MASTER_SSH_HOST\" - path: \"$TEMP_PATH\" - error: \"$ERROR_CODE\") - please send a bug report"
return
elif [[ "$TEMP_STRG" == "remote" ]]; then
exit_critical "Unable to check temp file/directory (host: \"$MASTER_SSH_HOST\" -> \"$REMOTE_SSH_HOST\" - path: \"$TEMP_PATH\" - error: \"$ERROR_CODE\") - please send a bug report"
return
fi
fi
# ---
if [[ "$ERROR_CODE" -eq 42 ]]; then
if [[ "$OPTION_DRYRUN" -eq 1 ]] && [[ "$TEMP_PATH" != *"remove-me"* ]]; then
if [[ "$TEMP_STRG" == "source" ]]; then
log_debug "Temp file/directory not found (path: \"$TEMP_PATH\" - error: \"$ERROR_CODE\") - ignoring (dry-run mode)"
elif [[ "$TEMP_STRG" == "master" ]]; then
log_debug "Temp file/directory not found (host: \"$MASTER_SSH_HOST\" - path: \"$TEMP_PATH\" - error: \"$ERROR_CODE\") - ignoring (dry-run mode)"
elif [[ "$TEMP_STRG" == "remote" ]]; then
log_debug "Temp file/directory not found (host: \"$MASTER_SSH_HOST\" -> \"$REMOTE_SSH_HOST\" - path: \"$TEMP_PATH\" - error: \"$ERROR_CODE\") - ignoring (dry-run mode)"
fi
else
if [[ "$TEMP_MODE" == "exists_soft" ]]; then
if [[ "$TEMP_STRG" == "source" ]]; then
# Note: ALL (check_temp_path: fallback)
log_warning "Temp file/directory not found (path: \"$TEMP_PATH\" - error: \"$ERROR_CODE\") - please send a bug report if this happens again"
elif [[ "$TEMP_STRG" == "master" ]]; then
# Note: ALL (check_temp_path: fallback)
log_warning "Temp file/directory not found (host: \"$MASTER_SSH_HOST\" - path: \"$TEMP_PATH\" - error: \"$ERROR_CODE\") - please send a bug report if this happens again"
elif [[ "$TEMP_STRG" == "remote" ]]; then
# Note: ALL (check_temp_path: fallback)
log_warning "Temp file/directory not found (host: \"$MASTER_SSH_HOST\" -> \"$REMOTE_SSH_HOST\" - path: \"$TEMP_PATH\" - error: \"$ERROR_CODE\") - please send a bug report if this happens again"
fi
elif [[ "$TEMP_MODE" == "exists_hard" ]]; then
if [[ "$TEMP_STRG" == "source" ]]; then
exit_error "Temp file/directory not found (path: \"$TEMP_PATH\" - error: \"$ERROR_CODE\") - aborting"
return
elif [[ "$TEMP_STRG" == "master" ]]; then
exit_error "Temp file/directory not found (host: \"$MASTER_SSH_HOST\" - path: \"$TEMP_PATH\" - error: \"$ERROR_CODE\") - aborting"
return
elif [[ "$TEMP_STRG" == "remote" ]]; then
exit_error "Temp file/directory not found (host: \"$MASTER_SSH_HOST\" -> \"$REMOTE_SSH_HOST\" - path: \"$TEMP_PATH\" - error: \"$ERROR_CODE\") - aborting"
return
fi
fi
fi
fi
fi
return 0
fi
# --- Delete
if [[ "$TEMP_MODE" == "delete_soft" ]] || [[ "$TEMP_MODE" == "delete_hard" ]]; then
if [[ "$OPTION_DRYRUN" -eq 1 ]] && [[ "$TEMP_PATH" != *"remove-me"* ]]; then
if [[ "$TEMP_STRG" == "source" ]]; then
if [[ "$TEMP_TYPE" == "path" ]]; then
dry_run "if [[ -d \"$TEMP_PATH\" ]]; then $CMD_RMDIR $ARGS_CPMKMVRM \"$TEMP_PATH\"; else (exit 42); fi\""
elif [[ "$TEMP_TYPE" == "path" ]]; then
dry_run "if [[ -f \"$TEMP_PATH\" ]]; then $CMD_RM $ARGS_CPMKMVRM \"$TEMP_PATH\"; else (exit 42); fi\""
fi
elif [[ "$TEMP_STRG" == "master" ]]; then
if [[ "$TEMP_TYPE" == "path" ]]; then
dry_run "$MASTER_SSH_OPTIONS \"$MASTER_SSH_HOST\" \"if [ -d \\\"$TEMP_PATH\\\" ]; then $CMD_RMDIR $ARGS_CPMKMVRM \\\"$TEMP_PATH\\\"; else exit 42; fi\""
elif [[ "$TEMP_TYPE" == "file" ]]; then
dry_run "$MASTER_SSH_OPTIONS \"$MASTER_SSH_HOST\" \"if [ -f \\\"$TEMP_PATH\\\" ]; then $CMD_RM $ARGS_CPMKMVRM \\\"$TEMP_PATH\\\"; else exit 42; fi\""
fi
elif [[ "$TEMP_STRG" == "remote" ]]; then
if [[ "$TEMP_TYPE" == "path" ]]; then
dry_run "$MASTER_SSH_OPTIONS \"$MASTER_SSH_HOST\" \"$REMOTE_SSH_OPTIONS \\\"$REMOTE_SSH_HOST\\\" \\\"if [ -d \\\\\"$TEMP_PATH\\\\\" ]; then $CMD_RMDIR $ARGS_CPMKMVRM \\\\\"$TEMP_PATH\\\\\"; else exit 42; fi\\\"\""
elif [[ "$TEMP_TYPE" == "file" ]]; then
dry_run "$MASTER_SSH_OPTIONS \"$MASTER_SSH_HOST\" \"$REMOTE_SSH_OPTIONS \\\"$REMOTE_SSH_HOST\\\" \\\"if [ -f \\\\\"$TEMP_PATH\\\\\" ]; then $CMD_RM $ARGS_CPMKMVRM \\\\\"$TEMP_PATH\\\\\"; else exit 42; fi\\\"\""
fi
fi
else
set +o errexit; trap - ERR
if [[ "$TEMP_STRG" == "source" ]]; then
if [[ "$TEMP_TYPE" == "path" ]]; then
if [[ -d "$TEMP_PATH" ]]; then $CMD_RMDIR $ARGS_CPMKMVRM "$TEMP_PATH" 2>&1 | $CMD_TEE -a "$LOG_FILE"; else (exit 42); fi
elif [[ "$TEMP_TYPE" == "file" ]]; then
if [[ -f "$TEMP_PATH" ]]; then $CMD_RM $ARGS_CPMKMVRM "$TEMP_PATH" 2>&1 | $CMD_TEE -a "$LOG_FILE"; else (exit 42); fi
fi
elif [[ "$TEMP_STRG" == "master" ]]; then
if [[ "$TEMP_TYPE" == "path" ]]; then
# Note: ALL (check_temp_path: MASTER_SSH_HOST => single brackets)
$MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "if [ -d \"$TEMP_PATH\" ]; then $CMD_RMDIR $ARGS_CPMKMVRM \"$TEMP_PATH\"; else exit 42; fi" 2>&1 | $CMD_TEE -a "$LOG_FILE"
elif [[ "$TEMP_TYPE" == "file" ]]; then
# Note: ALL (check_temp_path: MASTER_SSH_HOST => single brackets)
$MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "if [ -f \"$TEMP_PATH\" ]; then $CMD_RM $ARGS_CPMKMVRM \"$TEMP_PATH\"; else exit 42; fi" 2>&1 | $CMD_TEE -a "$LOG_FILE"
fi
elif [[ "$TEMP_STRG" == "remote" ]]; then
if [[ "$TEMP_TYPE" == "path" ]]; then
# Note: ALL (check_temp_path: REMOTE_SSH_HOST => single brackets)
$MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "$REMOTE_SSH_OPTIONS \"$REMOTE_SSH_HOST\" \"if [ -d \\\"$TEMP_PATH\\\" ]; then $CMD_RMDIR $ARGS_CPMKMVRM \\\"$TEMP_PATH\\\"; else exit 42; fi\"" 2>&1 | $CMD_TEE -a "$LOG_FILE"
elif [[ "$TEMP_TYPE" == "file" ]]; then
# Note: ALL (check_temp_path: REMOTE_SSH_HOST => single brackets)
$MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "$REMOTE_SSH_OPTIONS \"$REMOTE_SSH_HOST\" \"if [ -f \\\"$TEMP_PATH\\\" ]; then $CMD_RM $ARGS_CPMKMVRM \\\"$TEMP_PATH\\\"; else exit 42; fi\"" 2>&1 | $CMD_TEE -a "$LOG_FILE"
fi
fi
local ERROR_CODE="$?"
set_trap "$TEMP_TRAP"
# --- Check error code
if [[ "$ERROR_CODE" -ne 0 ]]; then
if [[ "$ERROR_CODE" -eq 255 ]]; then
if [[ "$TEMP_STRG" == "master" ]]; then
exit_critical "SSH error: connection failure (host: \"$MASTER_SSH_HOST\" - error: \"$ERROR_CODE\") - aborting"
return
elif [[ "$TEMP_STRG" == "remote" ]]; then
exit_critical "SSH error: connection failure (host: \"$MASTER_SSH_HOST\" -> \"$REMOTE_SSH_HOST\" - error: \"$ERROR_CODE\") - aborting"
return
fi
fi
# ---
if [[ "$ERROR_CODE" -ne 42 ]]; then
if [[ "$TEMP_MODE" == "delete_soft" ]]; then
if [[ "$TEMP_STRG" == "source" ]]; then
# Note: ALL (check_temp_path: fallback)
log_warning "Unable to delete temp file/directory (path: \"$TEMP_PATH\" - error: \"$ERROR_CODE\") - please send a bug report if this happens again"
elif [[ "$TEMP_STRG" == "master" ]]; then
# Note: ALL (check_temp_path: fallback)
log_warning "Unable to delete temp file/directory (host: \"$MASTER_SSH_HOST\" - path: \"$TEMP_PATH\" - error: \"$ERROR_CODE\") - please send a bug report if this happens again"
elif [[ "$TEMP_STRG" == "remote" ]]; then
# Note: ALL (check_temp_path: fallback)
log_warning "Unable to delete temp file/directory (host: \"$MASTER_SSH_HOST\" -> \"$REMOTE_SSH_HOST\" - path: \"$TEMP_PATH\" - error: \"$ERROR_CODE\") - please send a bug report if this happens again"
fi
elif [[ "$TEMP_MODE" == "delete_hard" ]]; then
if [[ "$TEMP_STRG" == "source" ]]; then
exit_error "Unable to delete temp file/directory (path: \"$TEMP_PATH\" - error: \"$ERROR_CODE\") - aborting"
return
elif [[ "$TEMP_STRG" == "master" ]]; then
exit_error "Unable to delete temp file/directory (host: \"$MASTER_SSH_HOST\" - path: \"$TEMP_PATH\" - error: \"$ERROR_CODE\") - aborting"
return
elif [[ "$TEMP_STRG" == "remote" ]]; then
exit_error "Unable to delete temp file/directory (host: \"$MASTER_SSH_HOST\" -> \"$REMOTE_SSH_HOST\" - path: \"$TEMP_PATH\" - error: \"$ERROR_CODE\") - aborting"
return
fi
fi
fi
# ---
if [[ "$ERROR_CODE" -eq 42 ]]; then
if [[ "$TEMP_STRG" == "source" ]]; then
log_debug "Temp file/directory not found (path: \"$TEMP_PATH\" - error: \"$ERROR_CODE\") - skipping"
elif [[ "$TEMP_STRG" == "master" ]]; then
log_debug "Temp file/directory not found (host: \"$MASTER_SSH_HOST\" - path: \"$TEMP_PATH\" - error: \"$ERROR_CODE\") - skipping"
elif [[ "$TEMP_STRG" == "remote" ]]; then
log_debug "Temp file/directory not found (host: \"$MASTER_SSH_HOST\" -> \"$REMOTE_SSH_HOST\" - path: \"$TEMP_PATH\" - error: \"$ERROR_CODE\") - skipping"
fi
fi
fi
fi
return 0
fi
}
# ---
function create_storage {
set_trap "Unable to create ${1:-undefined} storage"
log_trace "--> Create ${1:-undefined} storage"
local FUNC_INIT="$($CMD_DATE "+%s%1N")"
# ---
local TEMP_STRG="${1:-undefined}"
if [[ "$TEMP_STRG" != "master" ]] && [[ "$TEMP_STRG" != "remote" ]]; then
exit_critical "Invalid create_storage call (strg: \"$TEMP_STRG\") - please send a bug report"
return
fi
# --- Set paths (local)
if [[ "$TEMP_STRG" == "master" ]]; then
# Note: ALL (create_storage: dry-run => directory required)
if [[ "$OPTION_DRYRUN" -eq 1 ]]; then
local TEMP_DIR="$SERVER_TMP_DIR/${CALLER_TASK}_${CONFIG_TASK}_${BACKUP_CTID}_storage.remove-me"
else
local TEMP_DIR="$SERVER_TMP_DIR/${CALLER_TASK}_${CONFIG_TASK}_${BACKUP_CTID}_storage.delete-me"
fi
# --- Check temp directory (clean-up)
if [[ -d "$TEMP_DIR/" ]]; then
log_notice "Temp directory already exists (path: \"$TEMP_DIR\") - cleaning up"
set +o errexit; trap - ERR
local TEMP_SUBDIR=""
for TEMP_SUBDIR in conf data diff dump hash meta null task; do
if [[ ! -d "$TEMP_DIR/$TEMP_SUBDIR/" ]]; then
continue
fi
$CMD_RMDIR $ARGS_CPMKMVRM "$TEMP_DIR/$TEMP_SUBDIR/" 2>&1 | $CMD_TEE -a "$LOG_FILE"
done
# --- Delete temp directory
$CMD_RMDIR $ARGS_CPMKMVRM "$TEMP_DIR/" 2>&1 | $CMD_TEE -a "$LOG_FILE"
local ERROR_CODE="$?"
set_trap "Unable to create $TEMP_STRG storage"
# --- Check error code
if [[ "$ERROR_CODE" -ne 0 ]]; then
exit_error "Unable to delete temp directory (path: \"$TEMP_DIR\" - error: \"$ERROR_CODE\") - aborting"
return
fi
fi
# --- Create temp directory
local TEMP_SUBDIR=""
for TEMP_SUBDIR in $TEMP_DIR $TEMP_DIR/conf $TEMP_DIR/data $TEMP_DIR/diff $TEMP_DIR/dump $TEMP_DIR/hash $TEMP_DIR/meta $TEMP_DIR/null $TEMP_DIR/task; do
set +o errexit; trap - ERR
# Note: ALL (create_storage: create "data" directory => 0750)
$CMD_MKDIR $ARGS_CPMKMVRM "$TEMP_SUBDIR/" 2>&1 | $CMD_TEE -a "$LOG_FILE" && \
$CMD_CHOWN $ARGS_CHOWNMOD "root:root" "$TEMP_SUBDIR/" 2>&1 | $CMD_TEE -a "$LOG_FILE" && \
$CMD_CHMOD $ARGS_CHOWNMOD "0750" "$TEMP_SUBDIR/" 2>&1 | $CMD_TEE -a "$LOG_FILE"
local ERROR_CODE="$?"
set_trap "Unable to create $TEMP_STRG storage"
# --- Check error code
if [[ "$ERROR_CODE" -ne 0 ]]; then
exit_error "Unable to create temp directory (path: \"$TEMP_SUBDIR\" - error: \"$ERROR_CODE\") - aborting"
return
fi
done
fi
# --- Create storage
if [[ "$OPTION_DRYRUN" -eq 1 ]]; then
if [[ "$TEMP_STRG" == "master" ]]; then
dry_run "$MASTER_SSH_OPTIONS \"$MASTER_SSH_HOST\" \" \
$CMD_MKDIR $ARGS_CPMKMVRM -m \\\"0750\\\" -p \\\"$MASTER_DIR_PATH/$BACKUP_CTID/\\\" && \
$CMD_CHOWN $ARGS_CHOWNMOD \\\"root:root\\\" \\\"$MASTER_DIR_PATH/$BACKUP_CTID/\\\" && \
$CMD_CHMOD $ARGS_CHOWNMOD \\\"0750\\\" \\\"$MASTER_DIR_PATH/$BACKUP_CTID/\\\" \
\""
elif [[ "$TEMP_STRG" == "remote" ]]; then
dry_run "$MASTER_SSH_OPTIONS \"$MASTER_SSH_HOST\" \"$REMOTE_SSH_OPTIONS \\\"$REMOTE_SSH_HOST\\\" \\\" \
$CMD_MKDIR $ARGS_CPMKMVRM -m \\\\\"0750\\\\\" -p \\\\\"$REMOTE_DIR_PATH/$BACKUP_CTID/\\\\\" && \
$CMD_CHOWN $ARGS_CHOWNMOD \\\\\"root:root\\\\\" \\\\\"$REMOTE_DIR_PATH/$BACKUP_CTID/\\\\\" && \
$CMD_CHMOD $ARGS_CHOWNMOD \\\\\"0750\\\\\" \\\\\"$REMOTE_DIR_PATH/$BACKUP_CTID/\\\\\" \
\\\"\""
fi
else
set +o errexit; trap - ERR
if [[ "$TEMP_STRG" == "master" ]]; then
$MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" " \
$CMD_MKDIR $ARGS_CPMKMVRM -m \"0750\" -p \"$MASTER_DIR_PATH/$BACKUP_CTID/\" && \
$CMD_CHOWN $ARGS_CHOWNMOD \"root:root\" \"$MASTER_DIR_PATH/$BACKUP_CTID/\" && \
$CMD_CHMOD $ARGS_CHOWNMOD \"0750\" \"$MASTER_DIR_PATH/$BACKUP_CTID/\" \
" 2>&1 | $CMD_TEE -a "$LOG_FILE"
elif [[ "$TEMP_STRG" == "remote" ]]; then
$MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "$REMOTE_SSH_OPTIONS \"$REMOTE_SSH_HOST\" \" \
$CMD_MKDIR $ARGS_CPMKMVRM -m \\\"0750\\\" -p \\\"$REMOTE_DIR_PATH/$BACKUP_CTID/\\\" && \
$CMD_CHOWN $ARGS_CHOWNMOD \\\"root:root\\\" \\\"$REMOTE_DIR_PATH/$BACKUP_CTID/\\\" && \
$CMD_CHMOD $ARGS_CHOWNMOD \\\"0750\\\" \\\"$REMOTE_DIR_PATH/$BACKUP_CTID/\\\" \
\"" 2>&1 | $CMD_TEE -a "$LOG_FILE"
fi
local ERROR_CODE="$?"
set_trap "Unable to create $TEMP_STRG storage"
# --- Check error code
if [[ "$ERROR_CODE" -ne 0 ]]; then
if [[ "$ERROR_CODE" -eq 255 ]]; then
if [[ "$TEMP_STRG" == "master" ]]; then
exit_critical "SSH error: connection failure (host: \"$MASTER_SSH_HOST\" - error: \"$ERROR_CODE\") - aborting"
return
elif [[ "$TEMP_STRG" == "remote" ]]; then
exit_critical "SSH error: connection failure (host: \"$MASTER_SSH_HOST\" -> \"$REMOTE_SSH_HOST\" - error: \"$ERROR_CODE\") - aborting"
return
fi
fi
if [[ "$TEMP_STRG" == "master" ]]; then
exit_error "Unable to create $TEMP_STRG storage (path: \"$MASTER_DIR_PATH\" - error: \"$ERROR_CODE\") - aborting"
return
elif [[ "$TEMP_STRG" == "remote" ]]; then
exit_error "Unable to create $TEMP_STRG storage (path: \"$REMOTE_DIR_PATH\" - error: \"$ERROR_CODE\") - aborting"
return
fi
fi
if [[ "$TEMP_STRG" == "master" ]]; then
log_info2 "Root path: \"$MASTER_DIR_PATH/$BACKUP_CTID\""
elif [[ "$TEMP_STRG" == "remote" ]]; then
log_info2 "Root path: \"$REMOTE_DIR_PATH/$BACKUP_CTID\""
fi
fi
# --- Set paths (rsync)
if [[ "$TEMP_STRG" == "master" ]]; then
local SOURCE_PATH="$TEMP_DIR"
local TARGET_PATH="$MASTER_SSH_PATH/$BACKUP_CTID"
check_temp_path "$SOURCE_PATH/" "source" "exists_hard" "Unable to create $TEMP_STRG storage"
elif [[ "$TEMP_STRG" == "remote" ]]; then
local SOURCE_PATH="$MASTER_DIR_PATH/$BACKUP_CTID"
local TARGET_PATH="$REMOTE_SSH_PATH/$BACKUP_CTID"
check_temp_path "$SOURCE_PATH/" "master" "exists_hard" "Unable to create $TEMP_STRG storage"
fi
# --- Set speed (rsync)
if [[ "$TEMP_STRG" == "master" ]]; then
set_speed "Creating $TEMP_STRG storage" "save" "$MASTER_RSYNC_UPLOAD_SPEED"
local SOURCE_RSYNC_COMMAND="$SPEED_SOURCE_RSYNC_COMMAND"
local MASTER_RSYNC_OPTIONS="$SPEED_MASTER_RSYNC_OPTIONS"
elif [[ "$TEMP_STRG" == "remote" ]]; then
set_speed "Creating $TEMP_STRG storage" "save" "$REMOTE_RSYNC_UPLOAD_SPEED"
local MASTER_RSYNC_COMMAND="$SPEED_MASTER_RSYNC_COMMAND"
local REMOTE_RSYNC_OPTIONS="$SPEED_REMOTE_RSYNC_OPTIONS"
fi
# --- Rsync options
if [[ "$TEMP_STRG" == "master" ]]; then
if [[ "$OPTION_DRYRUN" -eq 1 ]]; then
local SOURCE_RSYNC_COMMAND="$SOURCE_RSYNC_COMMAND --dry-run"
fi
if [[ "$OPTION_VERBOSE" -gt 1 ]]; then
local SOURCE_RSYNC_COMMAND="$SOURCE_RSYNC_COMMAND --progress"
fi
elif [[ "$TEMP_STRG" == "remote" ]]; then
if [[ "$OPTION_DRYRUN" -eq 1 ]]; then
local MASTER_RSYNC_COMMAND="$MASTER_RSYNC_COMMAND --dry-run"
fi
if [[ "$OPTION_VERBOSE" -gt 1 ]]; then
local MASTER_RSYNC_COMMAND="$MASTER_RSYNC_COMMAND --progress"
fi
fi
# --- Create storage
local TASK_INIT="$($CMD_DATE "+%s%1N")"
set +o errexit; trap - ERR SIGINT
if [[ "$TEMP_STRG" == "master" ]]; then
# Note: ALL (create_storage: directories => preserve modification time)
$SOURCE_RSYNC_COMMAND \
--rsh="$MASTER_SSH_OPTIONS" \
--rsync-path="$MASTER_RSYNC_OPTIONS" \
--timeout="$MASTER_RSYNC_TIMEOUT" \
--compress \
--compress-level="$MASTER_RSYNC_COMPRESSION_LEVEL" \
--archive \
--numeric-ids \
--no-times \
$MASTER_RSYNC_PARAMETERS \
$ARGS_RSYNC \
--include="/conf/" \
--include="/data/" \
--include="/diff/" \
--include="/dump/" \
--include="/hash/" \
--include="/meta/" \
--include="/null/" \
--include="/task/" \
--exclude="/**" \
"$SOURCE_PATH/" \
"$TARGET_PATH/" \
2>> "$LOG_FILE"
elif [[ "$TEMP_STRG" == "remote" ]]; then
# Note: ALL (create_storage: directories => preserve modification time)
$MASTER_SSH_OPTIONS $OPTION_TTY "$MASTER_SSH_HOST" "$MASTER_RSYNC_COMMAND \
--rsh=\"$REMOTE_SSH_OPTIONS\" \
--rsync-path=\"$REMOTE_RSYNC_OPTIONS\" \
--timeout=\"$REMOTE_RSYNC_TIMEOUT\" \
--compress \
--compress-level=\"$REMOTE_RSYNC_COMPRESSION_LEVEL\" \
--archive \
--numeric-ids \
--no-times \
$REMOTE_RSYNC_PARAMETERS \
$ARGS_RSYNC \
--include=\"/conf/\" \
--include=\"/data/\" \
--include=\"/diff/\" \
--include=\"/dump/\" \
--include=\"/hash/\" \
--include=\"/meta/\" \
--include=\"/null/\" \
--include=\"/task/\" \
--exclude=\"/**\" \
\"$SOURCE_PATH/\" \
\"$TARGET_PATH/\"" \
2>> "$LOG_FILE"
fi
local ERROR_CODE="$?"
set_trap "Unable to create $TEMP_STRG storage"
local TASK_STOP="$($CMD_DATE "+%s%1N")"
# --- Check temp directory (delete)
if [[ "$TEMP_STRG" == "master" ]]; then
# Note: ALL (create_storage: temp directory => should exist)
if [[ -d "$TEMP_DIR/" ]]; then
set +o errexit; trap - ERR
local TEMP_SUBDIR=""
for TEMP_SUBDIR in conf data diff dump hash meta null task; do
if [[ ! -d "$TEMP_DIR/$TEMP_SUBDIR/" ]]; then
continue
fi
$CMD_RMDIR $ARGS_CPMKMVRM "$TEMP_DIR/$TEMP_SUBDIR/" 2>&1 | $CMD_TEE -a "$LOG_FILE"
done
# --- Delete temp directory
$CMD_RMDIR $ARGS_CPMKMVRM "$TEMP_DIR/" 2>&1 | $CMD_TEE -a "$LOG_FILE"
local ERROR_CODE_2="$?"
set_trap "Unable to create $TEMP_STRG storage"
# --- Check error code 2
if [[ "$ERROR_CODE_2" -ne 0 ]]; then
# Note: ALL (create_storage: fallback)
log_warning "Unable to delete temp directory (path: \"$TEMP_DIR\" - error: \"$ERROR_CODE_2\") - please send a bug report if this happens again"
fi
fi
fi
# --- Check error code
if [[ "$ERROR_CODE" -ne 0 ]]; then
if [[ "$ERROR_CODE" -eq 255 ]]; then
if [[ "$TEMP_STRG" == "master" ]]; then
exit_critical "SSH error: connection failure (host: \"$MASTER_SSH_HOST\" - error: \"$ERROR_CODE\") - aborting"
return
elif [[ "$TEMP_STRG" == "remote" ]]; then
exit_critical "SSH error: connection failure (host: \"$MASTER_SSH_HOST\" -> \"$REMOTE_SSH_HOST\" - error: \"$ERROR_CODE\") - aborting"
return
fi
fi
if [[ "$ERROR_CODE" -eq 20 ]] || [[ "$ERROR_CODE" -eq 30 ]]; then
if [[ "$TEMP_STRG" == "master" ]]; then
exit_critical "Rsync error: received SIGUSR1, SIGINT or connection timed out (host: \"$MASTER_SSH_HOST\" - error: \"$ERROR_CODE\") - aborting"
return
elif [[ "$TEMP_STRG" == "remote" ]]; then
exit_critical "Rsync error: received SIGUSR1, SIGINT or connection timed out (host: \"$MASTER_SSH_HOST\" -> \"$REMOTE_SSH_HOST\" - error: \"$ERROR_CODE\") - aborting"
return
fi
fi
if [[ "$TEMP_STRG" == "master" ]]; then
exit_error "Unable to create $TEMP_STRG storage (path: \"$MASTER_DIR_PATH\" - error: \"$ERROR_CODE\") - aborting"
return
elif [[ "$TEMP_STRG" == "remote" ]]; then
exit_error "Unable to create $TEMP_STRG storage (path: \"$SOURCE_PATH\" - error: \"$ERROR_CODE\") - aborting"
return
fi
fi
# ---
local FUNC_STOP="$($CMD_DATE "+%s%1N")"
compute_time "$TASK_INIT" "$TASK_STOP"
if [[ "$TEMP_STRG" == "master" ]]; then
log_time "$DISPLAY_STAT_TIME_1ST - Create $TEMP_STRG storage (path: \"$MASTER_DIR_PATH/$BACKUP_CTID\")"
elif [[ "$TEMP_STRG" == "remote" ]]; then
log_time "$DISPLAY_STAT_TIME_1ST - Create $TEMP_STRG storage (path: \"$REMOTE_DIR_PATH/$BACKUP_CTID\")"
fi
log_trace "<-- Create $TEMP_STRG storage"
reset_trap
}
# ---
function delete_storage_path {
set_trap "Unable to delete files in ${1:-undefined} storage ${2:-undefined} directory"
log_trace "--> Delete files in ${1:-undefined} storage ${2:-undefined} directory"
local FUNC_INIT="$($CMD_DATE "+%s%1N")"
# ---
local TEMP_STRG="${1:-undefined}"
local TEMP_PATH="${2:-undefined}"
local TEMP_DATE="${3:-undefined}"
if [[ "$TEMP_STRG" != "master" ]] && [[ "$TEMP_STRG" != "remote" ]]; then
exit_critical "Invalid delete_storage_path call (strg: \"$TEMP_STRG\" - path: \"$TEMP_PATH\" - date: \"$TEMP_DATE\") - please send a bug report"
return
fi
if [[ "$TEMP_PATH" != "conf" ]] && [[ "$TEMP_PATH" != "data" ]] && [[ "$TEMP_PATH" != "diff" ]] && [[ "$TEMP_PATH" != "dump" ]] &&
[[ "$TEMP_PATH" != "hash" ]] && [[ "$TEMP_PATH" != "meta" ]] && [[ "$TEMP_PATH" != "null" ]] && [[ "$TEMP_PATH" != "task" ]] &&
[[ "$TEMP_PATH" != "lock" ]]; then
exit_critical "Invalid delete_storage_path call (strg: \"$TEMP_STRG\" - path: \"$TEMP_PATH\" - date: \"$TEMP_DATE\") - please send a bug report"
return
fi
if [[ "$TEMP_DATE" != "*" ]]; then
local DIFFBK_TS="${TEMP_DATE:0:4}${TEMP_DATE:5:2}${TEMP_DATE:8:2}${TEMP_DATE:11:2}${TEMP_DATE:14:2}${TEMP_DATE:17:2}"
if [[ "${#TEMP_DATE}" -ne 19 ]] || [[ "${#DIFFBK_TS}" -ne 14 ]] || [[ "$DIFFBK_TS" == *[!0-9]* ]] || [[ "$DIFFBK_TS" != "${TEMP_DATE//[\-_]/}" ]]; then
# [[ "$($CMD_DATE -d "${TEMP_DATE:0:4}-${TEMP_DATE:5:2}-${TEMP_DATE:8:2} ${TEMP_DATE:11:2}:${TEMP_DATE:14:2}:${TEMP_DATE:17:2}" "+%Y-%m-%d_%H-%M-%S")" != "$TEMP_DATE" ]]; then
exit_critical "Invalid delete_storage_path call (strg: \"$TEMP_STRG\" - path: \"$TEMP_PATH\" - date: \"$TEMP_DATE\") - please send a bug report"
return
fi
fi
# --- Set paths
if [[ "$TEMP_STRG" == "master" ]]; then
local SOURCE_PATH="$MASTER_DIR_PATH/$BACKUP_CTID/null"
local TARGET_PATH="$MASTER_DIR_PATH/$BACKUP_CTID/$TEMP_PATH"
elif [[ "$TEMP_STRG" == "remote" ]]; then
local SOURCE_PATH="$REMOTE_DIR_PATH/$BACKUP_CTID/null"
local TARGET_PATH="$REMOTE_DIR_PATH/$BACKUP_CTID/$TEMP_PATH"
fi
check_temp_path "$SOURCE_PATH/" "$TEMP_STRG" "exists_hard" "Unable to delete files in $TEMP_STRG storage $TEMP_PATH directory"
# --- Set speed (rsync)
if [[ "$TEMP_STRG" == "master" ]]; then
# FIXME (medium): MAGIC_NUMBER (MASTER_RSYNC_DELETE_SPEED / 6.00)
set_speed "Deleting $TEMP_STRG storage path" "scan" "$(( ( ( MASTER_RSYNC_DELETE_SPEED * 100 ) + 300 ) / 600 ))"
local RSYNC_COMMAND="$SPEED_MASTER_RSYNC_COMMAND"
# local RSYNC_OPTIONS="$SPEED_MASTER_RSYNC_OPTIONS"
elif [[ "$TEMP_STRG" == "remote" ]]; then
# FIXME (medium): MAGIC_NUMBER (REMOTE_RSYNC_DELETE_SPEED / 6.00)
set_speed "Deleting $TEMP_STRG storage path" "scan" "$(( ( ( REMOTE_RSYNC_DELETE_SPEED * 100 ) + 300 ) / 600 ))"
local RSYNC_COMMAND="$SPEED_REMOTE_RSYNC_COMMAND"
# local RSYNC_OPTIONS="$SPEED_REMOTE_RSYNC_OPTIONS"
fi
# --- Rsync options
if [[ "$OPTION_DRYRUN" -eq 1 ]]; then
local RSYNC_COMMAND="$RSYNC_COMMAND --dry-run"
fi
if [[ "$OPTION_VERBOSE" -gt 1 ]]; then
local RSYNC_COMMAND="$RSYNC_COMMAND --progress"
fi
# ---
if [[ "$TEMP_PATH" == "conf" ]]; then
local RSYNC_COMMAND="$RSYNC_COMMAND --include '/${TEMP_PATH}_${BACKUP_CTID}_${TEMP_DATE}.conf' --include '/${TEMP_PATH}_${BACKUP_CTID}_${TEMP_DATE}.start' --include '/${TEMP_PATH}_${BACKUP_CTID}_${TEMP_DATE}.stop' --include '/${TEMP_PATH}_${BACKUP_CTID}_${TEMP_DATE}.mount' --include '/${TEMP_PATH}_${BACKUP_CTID}_${TEMP_DATE}.umount' --include '/${TEMP_PATH}_${BACKUP_CTID}_${TEMP_DATE}.premount' --include '/${TEMP_PATH}_${BACKUP_CTID}_${TEMP_DATE}.postumount'"
# elif [[ "$TEMP_PATH" == "data" ]]; then
# local RSYNC_COMMAND="$RSYNC_COMMAND --include '/*'"
elif [[ "$TEMP_PATH" == "diff" ]]; then
local RSYNC_COMMAND="$RSYNC_COMMAND --include '/${TEMP_PATH}_${BACKUP_CTID}_${TEMP_DATE}/'"
elif [[ "$TEMP_PATH" == "dump" ]] || [[ "$TEMP_PATH" == "hash" ]]; then
local RSYNC_COMMAND="$RSYNC_COMMAND --include '/${TEMP_PATH}_${BACKUP_CTID}_${TEMP_DATE}.bz2'"
elif [[ "$TEMP_PATH" == "meta" ]] || [[ "$TEMP_PATH" == "task" ]]; then
local RSYNC_COMMAND="$RSYNC_COMMAND --include '/${TEMP_PATH}_${BACKUP_CTID}_${TEMP_DATE}.txt'"
# elif [[ "$TEMP_PATH" == "lock" ]] || [[ "$TEMP_PATH" == "null" ]]; then
# local RSYNC_COMMAND="$RSYNC_COMMAND --include ''"
fi
if [[ "$TEMP_PATH" == "diff" ]]; then
local RSYNC_COMMAND="$RSYNC_COMMAND --exclude '/*'"
elif [[ "$TEMP_PATH" != "data" ]]; then
local RSYNC_COMMAND="$RSYNC_COMMAND --exclude '/**'"
fi
# --- Delete storage path
local TASK_INIT="$($CMD_DATE "+%s%1N")"
set +o errexit; trap - ERR SIGINT
if [[ "$TEMP_STRG" == "master" ]]; then
# Note: ALL (delete_storage_path: local sync => compression is useless)
# Note: ALL (delete_storage_path: SOURCE_PATH is "null" directory => ignore modification time)
$MASTER_SSH_OPTIONS $OPTION_TTY "$MASTER_SSH_HOST" "$RSYNC_COMMAND \
--archive \
--delete \
--hard-links \
--numeric-ids \
--one-file-system \
--sparse \
--no-compress \
--no-times \
$MASTER_RSYNC_PARAMETERS \
$ARGS_RSYNC \
\"$SOURCE_PATH/\" \
\"$TARGET_PATH/\"" \
2>> "$LOG_FILE"
elif [[ "$TEMP_STRG" == "remote" ]]; then
# Note: ALL (delete_storage_path: local sync => compression is useless)
# Note: ALL (delete_storage_path: SOURCE_PATH is "null" directory => ignore modification time)
# Note: ALL (delete_storage_path: remote OPTION_TTY => STDERR "Shared connection to <host> closed")
$MASTER_SSH_OPTIONS $OPTION_TTY "$MASTER_SSH_HOST" "$REMOTE_SSH_OPTIONS $OPTION_TTY \"$REMOTE_SSH_HOST\" \"$RSYNC_COMMAND \
--archive \
--delete \
--hard-links \
--numeric-ids \
--one-file-system \
--sparse \
--no-compress \
--no-times \
$REMOTE_RSYNC_PARAMETERS \
$ARGS_RSYNC \
\\\"$SOURCE_PATH/\\\" \
\\\"$TARGET_PATH/\\\"\"" \
2>> "$LOG_FILE"
fi
local ERROR_CODE="$?"
set_trap "Unable to delete files in $TEMP_STRG storage $TEMP_PATH directory"
local TASK_STOP="$($CMD_DATE "+%s%1N")"
# --- Check error code
if [[ "$ERROR_CODE" -ne 0 ]]; then
if [[ "$ERROR_CODE" -eq 255 ]]; then
if [[ "$TEMP_STRG" == "master" ]]; then
exit_critical "SSH error: connection failure (host: \"$MASTER_SSH_HOST\" - error: \"$ERROR_CODE\") - aborting"
return
elif [[ "$TEMP_STRG" == "remote" ]]; then
exit_critical "SSH error: connection failure (host: \"$MASTER_SSH_HOST\" -> \"$REMOTE_SSH_HOST\" - error: \"$ERROR_CODE\") - aborting"
return
fi
fi
if [[ "$ERROR_CODE" -eq 20 ]] || [[ "$ERROR_CODE" -eq 30 ]]; then
if [[ "$TEMP_STRG" == "master" ]]; then
exit_critical "Rsync error: received SIGUSR1, SIGINT or connection timed out (host: \"$MASTER_SSH_HOST\" - error: \"$ERROR_CODE\") - aborting"
return
elif [[ "$TEMP_STRG" == "remote" ]]; then
exit_critical "Rsync error: received SIGUSR1, SIGINT or connection timed out (host: \"$MASTER_SSH_HOST\" -> \"$REMOTE_SSH_HOST\" - error: \"$ERROR_CODE\") - aborting"
return
fi
fi
exit_error "Unable to delete files in $TEMP_STRG storage $TEMP_PATH directory (path: \"$TARGET_PATH\" - date: \"$TEMP_DATE\" - error: \"$ERROR_CODE\") - aborting"
return
fi
# ---
if [[ "$TEMP_DATE" == "*" ]]; then
check_temp_path "$TARGET_PATH/" "$TEMP_STRG" "delete_hard" "Unable to delete files in $TEMP_STRG storage $TEMP_PATH directory"
if [[ "$TEMP_PATH" == "lock" ]]; then
if [[ "$TEMP_STRG" == "master" ]]; then
# Note: mutable
unset MASTER_LOCK
elif [[ "$TEMP_STRG" == "remote" ]]; then
# Note: mutable
unset REMOTE_LOCK
fi
fi
fi
log_info2 "${TEMP_PATH^} path: \"$TARGET_PATH\""
# ---
local FUNC_STOP="$($CMD_DATE "+%s%1N")"
compute_time "$TASK_INIT" "$TASK_STOP"
log_time "$DISPLAY_STAT_TIME_1ST - Delete files in $TEMP_STRG storage $TEMP_PATH directory (path: \"$TARGET_PATH\" - date: \"$TEMP_DATE\")"
log_trace "<-- Delete files in $TEMP_STRG storage $TEMP_PATH directory"
reset_trap
}
# ---
function compute_wait {
if [[ -z "$OPTION_WAIT" ]] || [[ -n "${WAIT_LOCK:-}" ]]; then
return 0
fi
if [[ "$CONFIG_TASK" == "$CALLER_TASK" ]] && ( [[ "$BATCH_CTID" != "all" ]] || [[ "${#OPTION_WAIT}" -ne 19 ]] ); then
return 0
elif [[ "$CONFIG_TASK" != "$CALLER_TASK" ]] && [[ "$BATCH_CTID" == "all" ]] && [[ "${#OPTION_WAIT}" -eq 19 ]]; then
return 0
fi
local WAIT_MODE="${1:-undefined}"
# --- datetime vs modulo
local TEMP_TIME="$($CMD_DATE "+%s%1N")"
if [[ "${#OPTION_WAIT}" -eq 19 ]]; then
local DATETIME="$OPTION_WAIT"
local TEMP_WAIT="$($CMD_DATE -d "${DATETIME:0:4}-${DATETIME:5:2}-${DATETIME:8:2} ${DATETIME:11:2}:${DATETIME:14:2}:${DATETIME:17:2}" "+%s%1N")"
else
local DATETIME="$($CMD_DATE -d "@$(( $($CMD_DATE -d "now $OPTION_WAIT seconds" "+%s") / $OPTION_WAIT * $OPTION_WAIT ))" "+%Y-%m-%d_%H-%M-%S")"
local TEMP_WAIT="$($CMD_DATE -d "${DATETIME:0:4}-${DATETIME:5:2}-${DATETIME:8:2} ${DATETIME:11:2}:${DATETIME:14:2}:${DATETIME:17:2}" "+%s%1N")"
# ---
# FIXME (low): hardcoded (datetime <= 10 seconds && modulo <= 300 seconds)
if [[ "$WAIT_MODE" == "pause" ]] && [[ "$TEMP_TIME" -ge "$(( TEMP_WAIT - 100 ))" ]] && [[ "$OPTION_WAIT" -le 300 ]]; then
log_info2 "Deferred start (datetime - $DATETIME): $(( ( TEMP_TIME - TEMP_WAIT + 100 ) / 10 )).$(( ( TEMP_TIME - TEMP_WAIT + 100 ) % 10 ))s too late - retrying (modulo: $OPTION_WAIT seconds)"
local DATETIME="$($CMD_DATE -d "@$(( $($CMD_DATE -d "now $(( OPTION_WAIT + 10 )) seconds" "+%s") / $OPTION_WAIT * $OPTION_WAIT ))" "+%Y-%m-%d_%H-%M-%S")"
local TEMP_WAIT="$($CMD_DATE -d "${DATETIME:0:4}-${DATETIME:5:2}-${DATETIME:8:2} ${DATETIME:11:2}:${DATETIME:14:2}:${DATETIME:17:2}" "+%s%1N")"
fi
fi
# ---
if [[ "$WAIT_MODE" == "pause" ]]; then
local TEMP_WAIT="$(( TEMP_WAIT - 100 ))"
elif [[ "$WAIT_MODE" == "start" ]]; then
readonly WAIT_LOCK="$CONFIG_TASK"
# else
# exit_critical "Invalid compute_wait call (mode: \"$WAIT_MODE\") - please send a bug report"
# return
fi
# ---
if [[ "$TEMP_TIME" -gt "$TEMP_WAIT" ]]; then
if [[ "$WAIT_MODE" == "pause" ]]; then
log_info2 "Deferred start (step 1/2 - $DATETIME): $(( ( TEMP_TIME - TEMP_WAIT ) / 10 )).$(( ( TEMP_TIME - TEMP_WAIT ) % 10 ))s too late - ignoring"
elif [[ "$WAIT_MODE" == "start" ]]; then
log_notice "Deferred start (step 2/2 - $DATETIME): $(( ( TEMP_TIME - TEMP_WAIT ) / 10 )).$(( ( TEMP_TIME - TEMP_WAIT ) % 10 ))s too late - ignoring (please, do not panic)"
fi
return 0
fi
# ---
if [[ "$WAIT_MODE" == "pause" ]]; then
log_info2 "Deferred start (step 1/2 - $DATETIME): waiting $(( ( TEMP_WAIT - TEMP_TIME ) / 10 )).$(( ( TEMP_WAIT - TEMP_TIME ) % 10 ))s"
elif [[ "$WAIT_MODE" == "start" ]]; then
log_info2 "Deferred start (step 2/2 - $DATETIME): waiting $(( ( TEMP_WAIT - TEMP_TIME ) / 10 )).$(( ( TEMP_WAIT - TEMP_TIME ) % 10 ))s"
# else
# exit_critical "Invalid compute_wait call (mode: \"$WAIT_MODE\") - please send a bug report"
# return
fi
# local TEMP_WAIT="$($CMD_DATE -d "${DATETIME:0:4}-${DATETIME:5:2}-${DATETIME:8:2} ${DATETIME:11:2}:${DATETIME:14:2}:${DATETIME:17:2}" "+%s%1N")"
local TEMP_TIME="$($CMD_DATE "+%s%1N")"
if [[ "$TEMP_WAIT" -gt "$TEMP_TIME" ]]; then
$CMD_SLEEP "$(( ( TEMP_WAIT - TEMP_TIME ) / 10 )).$(( ( TEMP_WAIT - TEMP_TIME ) % 10 ))"
fi
}
# ---
function compute_time {
local STAT_INIT_1ST="$1"
local STAT_STOP_1ST="$2"
# ---
local STAT_TIME_1ST="0.0s"
local EXEC_TIME_1ST="0s"
# ---
local STAT_DIFF_1ST="$(( STAT_STOP_1ST - STAT_INIT_1ST ))"
if [[ "$STAT_DIFF_1ST" -gt 0 ]]; then
local STAT_TIME_1ST="$(( STAT_DIFF_1ST / 864000 ))d$($CMD_PRINTF "%2d" "$(( ( STAT_DIFF_1ST % 864000 ) / 36000 ))")h$($CMD_PRINTF "%2d" "$(( ( ( STAT_DIFF_1ST % 864000 ) % 36000 ) / 600 ))")m$($CMD_PRINTF "%2d" "$(( ( ( ( STAT_DIFF_1ST % 864000 ) % 36000 ) % 600 ) / 10 ))").$(( ( ( ( ( STAT_DIFF_1ST % 864000 ) % 36000 ) % 600 ) % 10 ) ))s"
local STAT_TIME_1ST="${STAT_TIME_1ST#0d}"
local STAT_TIME_1ST="${STAT_TIME_1ST# 0h}"
local STAT_TIME_1ST="${STAT_TIME_1ST# 0m}"
local STAT_TIME_1ST="${STAT_TIME_1ST# }"
local STAT_TIME_1ST="${STAT_TIME_1ST// /0}"
local EXEC_TIME_1ST="${STAT_TIME_1ST%%.*}s"
fi
# ---
# Note: mutable
pad_left 10 "DISPLAY_STAT_TIME_1ST" "$STAT_TIME_1ST"
pad_left 9 "DISPLAY_EXEC_TIME_1ST" "$EXEC_TIME_1ST"
}
# ---
function check_global_config {
set_trap "Unable to check global configuration"
log_trace "--> Check global configuration"
# --- Cache
if [[ "$CONFIG_TASK" != "$CALLER_TASK" ]]; then
local TEMP_NAME=""
for TEMP_NAME in CONFIG_HOST CONFIG_TASK CONFIG_PATH CONFIG_DATE CONFIG_TYPE CONFIG_XVER; do
readonly $TEMP_NAME="${!TEMP_NAME}"
done
log_trace "<-- Check global configuration"
reset_trap
return 0
fi
# --- Configuration type
local FUNC_INIT="$($CMD_DATE "+%s%1N")"
# Note: PVE2/PVE3 (check_global_config: pveversion => pve-manager version 2/3)
if [[ -n "$CMD_PVECTL" ]]; then
CONFIG_TYPE="Proxmox"
if [[ -n "$CMD_PVEVERSION" ]] && [[ -n "$($CMD_PVEVERSION 2>> "$LOG_FILE" | $CMD_GREP "^pve\-manager/2\.")" ]]; then
CONFIG_XVER="2"
elif [[ -n "$CMD_PVEVERSION" ]] && [[ -n "$($CMD_PVEVERSION 2>> "$LOG_FILE" | $CMD_GREP "^pve\-manager/3\.")" ]]; then
CONFIG_XVER="3"
else
exit_critical "Unable to check $CONFIG_TYPE version - this tool only supports PVE2 & PVE3 - please send a bug report"
return
fi
# Note: PCS6/VZ7 (check_global_config: prlctl => version 6/7)
elif [[ -n "$CMD_PRLCTL" ]]; then
CONFIG_TYPE="Virtuozzo"
if [[ -n "$($CMD_PRLCTL --version 2>> "$LOG_FILE" | $CMD_GREP "^prlctl version 6\.")" ]]; then
CONFIG_XVER="6"
elif [[ -n "$($CMD_PRLCTL --version 2>> "$LOG_FILE" | $CMD_GREP "^prlctl version 7\.")" ]]; then
CONFIG_XVER="7"
else
exit_critical "Unable to check $CONFIG_TYPE version - this tool only supports PCS6 & VZ7 - please send a bug report"
return
fi
# Note: OPENVZ (check_global_config: vzctl => Legacy)
elif [[ -n "$CMD_VZCTL" ]]; then
CONFIG_TYPE="OpenVZ"
CONFIG_XVER=""
# Note: ALL (check_global_config: /etc/os-release)
elif [[ -f "/etc/os-release" ]]; then
CONFIG_TYPE="$($CMD_GREP "^ID=" "/etc/os-release" 2>> "$LOG_FILE" | $CMD_CUT -d "=" -f 2 -s | $CMD_TR -d '"')"
CONFIG_TYPE="${CONFIG_TYPE,}"
CONFIG_TYPE="${CONFIG_TYPE:-Unknown}"
CONFIG_XVER="$($CMD_GREP "^VERSION_ID=" "/etc/os-release" 2>> "$LOG_FILE" | $CMD_CUT -d "=" -f 2 -s | $CMD_TR -d '"')"
CONFIG_XVER="${CONFIG_XVER:-Unknown}"
else
CONFIG_TYPE="Unknown"
CONFIG_XVER="Unknown"
fi
# --- Check settings
local TEMP_NAME=""
for TEMP_NAME in CONFIG_HOST CONFIG_TASK CONFIG_PATH CONFIG_DATE CONFIG_TYPE CONFIG_XVER; do
if [[ -z "${!TEMP_NAME}" ]] && [[ "$TEMP_NAME" != "CONFIG_XVER" ]]; then
exit_critical "Unexpected configuration setting (name: \"$TEMP_NAME\" - value: \"${!TEMP_NAME}\") - please send a bug report"
return
fi
readonly $TEMP_NAME="${!TEMP_NAME}"
export $TEMP_NAME
done
# --- Check config type
if [[ "$CONFIG_TYPE" != "OpenVZ" ]] && [[ "$CONFIG_TYPE" != "Virtuozzo" ]] && [[ "$CONFIG_TYPE" != "Proxmox" ]]; then
if [[ "$BATCH_TASK" == "backup" ]] || ( [[ "$BATCH_TASK" == "restore" ]] && [[ "$CALLER_ARGS" != "$BATCH_TASK ${BACKUP_CTID:-$BATCH_CTID} list"* ]] ); then
exit_error "Unable to detect server configuration type (OpenVZ/Virtuozzo/Proxmox) - aborting"
return
fi
log_info2 "Unable to detect server configuration type (OpenVZ/Virtuozzo/Proxmox) - ignoring"
fi
# --- Check OpenVZ tools
if [[ -z "$CMD_VZCTL" ]] || [[ -z "$CMD_VZLIST" ]] || [[ -z "$CMD_VZPID" ]]; then
if [[ "$BATCH_TASK" == "backup" ]] || ( [[ "$BATCH_TASK" == "restore" ]] && [[ "$CALLER_ARGS" != "$BATCH_TASK ${BACKUP_CTID:-$BATCH_CTID} list"* ]] ); then
exit_error "Unable to detect OpenVZ tools (vzctl/vzlist/vzpid) - aborting"
return
fi
log_info2 "Unable to detect OpenVZ tools (vzctl/vzlist/vzpid) - ignoring"
fi
# ---
local FUNC_STOP="$($CMD_DATE "+%s%1N")"
compute_time "$FUNC_INIT" "$FUNC_STOP"
log_time "$DISPLAY_STAT_TIME_1ST - Check global configuration"
log_trace "<-- Check global configuration"
reset_trap
}
# ---
function check_global_options {
set_trap "Unable to check global options"
log_trace "--> Check global options"
# --- Cache
if [[ "$CONFIG_TASK" != "$CALLER_TASK" ]]; then
local TEMP_NAME=""
for TEMP_NAME in OPTION_BACKTRACE OPTION_CONFIG OPTION_DRYRUN OPTION_EXPERIMENTAL OPTION_FORCE OPTION_HOSTNAME OPTION_IDLETIME OPTION_KEEP OPTION_LOGLEVEL OPTION_MAXIMUM OPTION_OPTIMIZE OPTION_PAUSE OPTION_QUIET OPTION_RESYNC OPTION_STORAGE OPTION_TURBO OPTION_UNLOCK OPTION_VERBOSE OPTION_XTRACE OPTION_WAIT; do
readonly $TEMP_NAME="${!TEMP_NAME}"
done
for TEMP_NAME in ARGS_BZIP2 ARGS_CHOWNMOD ARGS_CPMKMVRM ARGS_LVM2 ARGS_MOUNT ARGS_RSYNC ARGS_TAR ARGS_VZCTL; do
readonly $TEMP_NAME="${!TEMP_NAME}"
done
log_trace "<-- Check global options"
reset_trap
return 0
fi
# --- Set defaults
local FUNC_INIT="$($CMD_DATE "+%s%1N")"
# Note: ALL (check_global_options: OPTION_* => readonly later)
OPTION_BACKTRACE="0"
OPTION_CONFIG=""
OPTION_DRYRUN="0"
OPTION_EXPERIMENTAL="0"
OPTION_FORCE="0"
OPTION_HOSTNAME=""
OPTION_IDLETIME=""
OPTION_KEEP=""
OPTION_LOGLEVEL="7"
OPTION_MAXIMUM=""
OPTION_OPTIMIZE="0"
OPTION_PAUSE="0"
OPTION_QUIET="0"
OPTION_RESYNC="0"
OPTION_STORAGE=""
OPTION_TURBO="0"
OPTION_UNLOCK="0"
OPTION_VERBOSE="0"
OPTION_XTRACE="0"
OPTION_WAIT=""
# --- Parse options
local OPTERR="1"
local OPTVAR=""
local OPTIND=""
local OPTARG=""
# ---
while [[ "$#" -gt 0 ]]; do
while getopts "bc:defh:i:k:l:m:o:pqrs:tuvw:x" OPTVAR; do
if [[ "$OPTVAR" == "b" ]]; then
OPTION_BACKTRACE="1"
elif [[ "$OPTVAR" == "c" ]]; then
if [[ -z "$OPTARG" ]]; then
exit_error "OPTION_CONFIG value is unset (option: \"-$OPTVAR\" - value: \"$OPTARG\") - aborting"
return
fi
if [[ -f "$CONFIG_PATH/etc/$OPTARG" ]]; then
OPTION_CONFIG="$($CMD_READLINK -e -n "$CONFIG_PATH/etc/$OPTARG" || true)"
elif [[ -f "$OPTARG" ]]; then
OPTION_CONFIG="$($CMD_READLINK -e -n "$OPTARG" || true)"
else
exit_error "$CALLER_TASK configuration file not found (path: \"$OPTARG\") - aborting"
return
fi
if [[ -z "$OPTION_CONFIG" ]]; then
exit_critical "Unable to check configuration file (path: \"$OPTARG\" - real: \"$OPTION_CONFIG\") - please send a bug report"
return
fi
elif [[ "$OPTVAR" == "d" ]]; then
OPTION_DRYRUN="1"
elif [[ "$OPTVAR" == "e" ]]; then
OPTION_EXPERIMENTAL="1"
elif [[ "$OPTVAR" == "f" ]]; then
OPTION_FORCE="1"
elif [[ "$OPTVAR" == "h" ]]; then
if [[ -z "$OPTARG" ]]; then
exit_error "OPTION_HOSTNAME value is unset (option: \"-$OPTVAR\" - value: \"$OPTARG\") - aborting"
return
fi
OPTION_HOSTNAME="$OPTARG"
elif [[ "$OPTVAR" == "i" ]]; then
if [[ -z "$OPTARG" ]]; then
exit_error "OPTION_IDLETIME value is unset (option: \"-$OPTVAR\" - value: \"$OPTARG\") - aborting"
return
fi
if [[ ! "$OPTARG" =~ ^[0-9]+\-[0-9]+$ ]] || [[ "${OPTARG%%-*}" -gt 23 ]] || [[ "${OPTARG#*-}" -gt 23 ]] || [[ "${OPTARG%%-*}" == "${OPTARG#*-}" ]]; then
exit_error "OPTION_IDLETIME value is not a valid range: <start>-<end> (option: \"-$OPTVAR\" - value: \"$OPTARG\") - aborting"
return
fi
OPTION_IDLETIME="$OPTARG"
elif [[ "$OPTVAR" == "k" ]]; then
if [[ -z "$OPTARG" ]]; then
exit_error "OPTION_KEEP value is unset (option: \"-$OPTVAR\" - value: \"$OPTARG\") - aborting"
return
fi
if [[ "$OPTARG" == *[!0-9]* ]] || [[ "$OPTARG" -gt 365 ]]; then
exit_error "OPTION_KEEP value is not a valid integer: 0-365 (option: \"-$OPTVAR\" - value: \"$OPTARG\") - aborting"
return
fi
OPTION_KEEP="$OPTARG"
elif [[ "$OPTVAR" == "l" ]]; then
if [[ -z "$OPTARG" ]]; then
exit_error "OPTION_LOGLEVEL value is unset (option: \"-$OPTVAR\" - value: \"$OPTARG\") - aborting"
return
fi
if [[ "$OPTARG" == *[!0-9]* ]] || [[ "$OPTARG" -gt 11 ]]; then
exit_error "OPTION_LOGLEVEL value is not a valid integer: 0-11 (option: \"-$OPTVAR\" - value: \"$OPTARG\") - aborting"
return
fi
OPTION_LOGLEVEL="$OPTARG"
elif [[ "$OPTVAR" == "m" ]]; then
if [[ -z "$OPTARG" ]]; then
exit_error "OPTION_MAXIMUM value is unset (option: \"-$OPTVAR\" - value: \"$OPTARG\") - aborting"
return
fi
if [[ "$OPTARG" == *[!0-9]* ]]; then
exit_error "OPTION_MAXIMUM value is not a valid integer: 0-2147483647 (option: \"-$OPTVAR\" - value: \"$OPTARG\") - aborting"
return
fi
OPTION_MAXIMUM="$OPTARG"
elif [[ "$OPTVAR" == "o" ]]; then
if [[ -z "$OPTARG" ]]; then
exit_error "OPTION_OPTIMIZE value is unset (option: \"-$OPTVAR\" - value: \"$OPTARG\") - aborting"
return
fi
if [[ "$OPTARG" == *[!0-9]* ]] || [[ "$OPTARG" -gt 63 ]]; then
exit_error "OPTION_OPTIMIZE value is not a valid integer: 0-63 (option: \"-$OPTVAR\" - value: \"$OPTARG\") - aborting"
return
fi
OPTION_OPTIMIZE="$OPTARG"
elif [[ "$OPTVAR" == "p" ]]; then
OPTION_PAUSE="1"
elif [[ "$OPTVAR" == "q" ]]; then
if [[ "$OPTION_QUIET" -lt 3 ]]; then
OPTION_QUIET="$(( OPTION_QUIET + 1 ))"
fi
elif [[ "$OPTVAR" == "r" ]]; then
if [[ "$OPTION_RESYNC" -lt 2 ]]; then
OPTION_RESYNC="$(( OPTION_RESYNC + 1 ))"
fi
elif [[ "$OPTVAR" == "s" ]]; then
if [[ -z "$OPTARG" ]]; then
exit_error "OPTION_STORAGE value is unset (option: \"-$OPTVAR\" - value: \"$OPTARG\") - aborting"
return
fi
# ---
# Note: PVE2/PVE3 (check_global_options: PVE_STORAGE_LIST => CMD_PVESM status)
if [[ "$CONFIG_TYPE" == "Proxmox" ]]; then
local PVE_STORAGE_LIST="$($CMD_PVESM status 2>> "$LOG_FILE" | $CMD_CUT -d " " -f "1" -s)"
# local PVE_STORAGE_LIST="${PVE_STORAGE_LIST//$'\r'/ }"
local PVE_STORAGE_LIST="${PVE_STORAGE_LIST//$'\n'/ }"
# local PVE_STORAGE_LIST="${PVE_STORAGE_LIST// / }"
if [[ -z "${PVE_STORAGE_LIST// /}" ]]; then
exit_critical "Unable to check $CONFIG_TYPE ${CONFIG_XVER:-Legacy} storage list (list: \"$PVE_STORAGE_LIST\") - please send a bug report"
return
fi
# ---
if [[ ! "$OPTARG" =~ ^[0-9a-f\-_\.]+$ ]]; then
exit_error "Invalid $CONFIG_TYPE ${CONFIG_XVER:-Legacy} storage name (name: \"$OPTARG\" - list: \"$PVE_STORAGE_LIST\") - aborting"
return
fi
# ---
local PVE_STORAGE_TEMP="$($CMD_PVESM status 2>> "$LOG_FILE" | $CMD_TR -s " " | $CMD_GREP "^${OPTARG} ")"
local PVE_STORAGE_NAME="${PVE_STORAGE_TEMP%% *}"
local PVE_STORAGE_TYPE="${PVE_STORAGE_TEMP#* }"
if [[ -z "$PVE_STORAGE_NAME" ]] || [[ "$PVE_STORAGE_NAME" != "$OPTARG" ]]; then
exit_error "$CONFIG_TYPE ${CONFIG_XVER:-Legacy} storage not found (name: \"$OPTARG\" - type: \"$PVE_STORAGE_TYPE\" - list: \"$PVE_STORAGE_LIST\") - aborting"
return
fi
if [[ -z "$PVE_STORAGE_TYPE" ]] || [[ "$PVE_STORAGE_TYPE" != "dir" ]]; then
exit_error "$CONFIG_TYPE ${CONFIG_XVER:-Legacy} storage is not a directory (name: \"$OPTARG\" - type: \"$PVE_STORAGE_TYPE\" - list: \"$PVE_STORAGE_LIST\") - aborting"
return
fi
readonly TARGET_STORAGE_NAME="$OPTARG"
export TARGET_STORAGE_NAME
# ---
# FIXME (low): hardcoded (PVE2/PVE3 - /etc/pve/storage.cfg)
local PVE_STORAGE_FILE="/etc/pve/storage.cfg"
if [[ ! -f "$PVE_STORAGE_FILE" ]]; then
# FIXME (low): hardcoded (PVE2/PVE3 - "local" storage)
if [[ "$TARGET_STORAGE_NAME" == "local" ]]; then
continue
fi
# ---
exit_critical "$CONFIG_TYPE ${CONFIG_XVER:-Legacy} storage configuration file not found (path: \"$PVE_STORAGE_FILE\" - name: \"$OPTARG\" - type: \"$PVE_STORAGE_TYPE\" - list: \"$PVE_STORAGE_LIST\") - please send a bug report"
return
fi
# ---
local PVE_STORAGE_PATH="$($CMD_GREP -A "1" "^dir: ${TARGET_STORAGE_NAME}$" "$PVE_STORAGE_FILE" 2>> "$LOG_FILE" | $CMD_TR -d "\t" | $CMD_GREP "^path " | $CMD_CUT -d " " -f "2" -s)"
if [[ -z "$PVE_STORAGE_PATH" ]] || [[ ! -d "$PVE_STORAGE_PATH/" ]]; then
exit_critical "Unable to check $CONFIG_TYPE ${CONFIG_XVER:-Legacy} storage directory (file: \"$PVE_STORAGE_FILE\" - name: \"$TARGET_STORAGE_NAME\" - path: \"$PVE_STORAGE_PATH\") - please send a bug report"
return
fi
# ---
readonly TARGET_STORAGE_PATH="$($CMD_READLINK -e -n "$PVE_STORAGE_PATH/private/" || true)"
export TARGET_STORAGE_PATH
if [[ -z "$TARGET_STORAGE_PATH" ]] || [[ ! -d "$TARGET_STORAGE_PATH/" ]]; then
exit_critical "Unable to check $CONFIG_TYPE ${CONFIG_XVER:-Legacy} storage directory (file: \"/etc/pve/storage.cfg\" - name: \"$TARGET_STORAGE_NAME\" - path: \"$PVE_STORAGE_PATH/private\" - real: \"$TARGET_STORAGE_PATH\") - please send a bug report"
return
fi
OPTION_STORAGE="$OPTARG"
else
readonly TARGET_STORAGE_NAME=""
export TARGET_STORAGE_NAME
if [[ ! -d "$OPTARG/" ]]; then
exit_error "Storage directory not found (path: \"$OPTARG\") - aborting"
return
fi
# ---
readonly TARGET_STORAGE_PATH="$($CMD_READLINK -e -n "$OPTARG" || true)"
export TARGET_STORAGE_PATH
if [[ -z "$TARGET_STORAGE_PATH" ]]; then
exit_critical "Unable to check $CONFIG_TYPE ${CONFIG_XVER:-Legacy} storage directory (path: \"$OPTARG\" - real: \"$TARGET_STORAGE_PATH\") - please send a bug report"
return
fi
OPTION_STORAGE="$OPTARG"
fi
elif [[ "$OPTVAR" == "t" ]]; then
if [[ "$OPTION_TURBO" -lt 3 ]]; then
OPTION_TURBO="$(( OPTION_TURBO + 1 ))"
fi
elif [[ "$OPTVAR" == "u" ]]; then
OPTION_UNLOCK="1"
elif [[ "$OPTVAR" == "v" ]]; then
if [[ "$OPTION_VERBOSE" -lt 3 ]]; then
OPTION_VERBOSE="$(( OPTION_VERBOSE + 1 ))"
fi
elif [[ "$OPTVAR" == "w" ]]; then
if [[ -z "$OPTARG" ]]; then
exit_error "OPTION_WAIT value is unset (option: \"-$OPTVAR\" - value: \"$OPTARG\") - aborting"
return
fi
# --- datetime vs modulo
if [[ "${#OPTARG}" -eq 19 ]]; then
local TEMP_WAIT=${OPTARG:0:4}${OPTARG:5:2}${OPTARG:8:2}${OPTARG:11:2}${OPTARG:14:2}${OPTARG:17:2}
if [[ "${#TEMP_WAIT}" -ne 14 ]] || [[ "$TEMP_WAIT" == *[!0-9]* ]] || [[ "$TEMP_WAIT" != "${OPTARG//[\-_]/}" ]] ||
[[ "$($CMD_DATE -d "${OPTARG:0:4}-${OPTARG:5:2}-${OPTARG:8:2} ${OPTARG:11:2}:${OPTARG:14:2}:${OPTARG:17:2}" "+%Y-%m-%d_%H-%M-%S")" != "$OPTARG" ]]; then
exit_error "OPTION_WAIT value is not a valid date: YYYY-MM-DD_HH-MM-SS (option: \"-$OPTVAR\" - value: \"$OPTARG\") - aborting"
return
fi
OPTION_WAIT="$OPTARG"
else
if [[ "$OPTARG" == *[!0-9]* ]] || [[ "$OPTARG" -gt 86400 ]] || ( [[ "$OPTARG" -gt 0 ]] && [[ "$OPTARG" -lt 10 ]] ); then
exit_error "OPTION_WAIT value is not a valid integer: 0 or 10-86400 (option: \"-$OPTVAR\" - value: \"$OPTARG\") - aborting"
return
fi
if [[ "$OPTARG" -eq 0 ]]; then
continue
fi
OPTION_WAIT="$OPTARG"
fi
elif [[ "$OPTVAR" == "x" ]]; then
OPTION_XTRACE="1"
else
exit_error "Invalid option (name: \"$OPTVAR\") - aborting"
return
fi
done
shift
done
# --- Set options
local TEMP_NAME=""
for TEMP_NAME in OPTION_BACKTRACE OPTION_CONFIG OPTION_DRYRUN OPTION_EXPERIMENTAL OPTION_FORCE OPTION_HOSTNAME OPTION_IDLETIME OPTION_KEEP OPTION_LOGLEVEL OPTION_MAXIMUM OPTION_OPTIMIZE OPTION_PAUSE OPTION_QUIET OPTION_RESYNC OPTION_STORAGE OPTION_TURBO OPTION_UNLOCK OPTION_VERBOSE OPTION_XTRACE OPTION_WAIT; do
readonly $TEMP_NAME="${!TEMP_NAME}"
export $TEMP_NAME
done
# ---
if [[ "$OPTION_VERBOSE" -gt 2 ]]; then
ARGS_BZIP2="-v -v"
ARGS_CHOWNMOD="-v"
ARGS_CPMKMVRM="-v"
ARGS_LVM2="-v -v"
ARGS_MOUNT="-v"
ARGS_RSYNC="--human-readable --human-readable --itemize-changes --itemize-changes --verbose --verbose --stats"
ARGS_TAR="-v"
ARGS_VZCTL="--verbose --verbose"
elif [[ "$OPTION_VERBOSE" -gt 1 ]]; then
ARGS_BZIP2="-v"
ARGS_CHOWNMOD=""
ARGS_CPMKMVRM=""
ARGS_LVM2="-v"
ARGS_MOUNT="-v"
ARGS_RSYNC="--human-readable --human-readable --itemize-changes --verbose --stats"
ARGS_TAR="-v"
ARGS_VZCTL="--verbose"
elif [[ "$OPTION_VERBOSE" -gt 0 ]]; then
ARGS_BZIP2=""
ARGS_CHOWNMOD=""
ARGS_CPMKMVRM=""
ARGS_LVM2=""
ARGS_MOUNT=""
ARGS_RSYNC="--human-readable --human-readable --itemize-changes"
ARGS_TAR=""
ARGS_VZCTL=""
else
ARGS_BZIP2=""
ARGS_CHOWNMOD=""
ARGS_CPMKMVRM=""
# Note: ALL (check_global_options: ARGS_LVM2=="--quiet" => not implemented)
ARGS_LVM2="--quiet"
ARGS_MOUNT=""
ARGS_RSYNC=""
ARGS_TAR=""
# Note: ALL (check_global_options: ARGS_VZCTL=="--quiet" => no output at all)
ARGS_VZCTL="--quiet"
fi
# ---
local TEMP_NAME=""
# Note: ALL (check_global_options: ARGS_CHOWNMOD/ARGS_CPMKMVRM => readonly later)
for TEMP_NAME in ARGS_BZIP2 ARGS_LVM2 ARGS_MOUNT ARGS_RSYNC ARGS_TAR ARGS_VZCTL; do
readonly $TEMP_NAME="${!TEMP_NAME}"
export $TEMP_NAME
done
# Note: ALL (check_global_options: ARGS_CHOWNMOD/ARGS_CPMKMVRM => update task)
if [[ "$CONFIG_TASK" == "$CALLER_TASK" ]] && [[ "$BATCH_TASK" == "update" ]]; then
readonly ARGS_CHOWNMOD="$ARGS_CHOWNMOD"
export ARGS_CHOWNMOD
readonly ARGS_CPMKMVRM="$ARGS_CPMKMVRM"
export ARGS_CPMKMVRM
fi
# --- Check vars
# $CMD_GREP -P -h -r "^\s*[0-9A-Z_]+=" lib/* | $CMD_CUT -d "=" -f "1" -s | $CMD_TR -s " " | $CMD_SORT -u
# ---
if [[ "$CONFIG_TASK" == "$CALLER_TASK" ]] && [[ "$OPTION_DRYRUN" -eq 1 ]]; then
log_notice "Dry run mode enabled - nothing will happen for real, I promise!"
task_pause 3 force
fi
if [[ "$CONFIG_TASK" == "$CALLER_TASK" ]] && [[ "$OPTION_EXPERIMENTAL" -eq 1 ]]; then
log_notice "Experimental mode has been removed, maybe in a near future!"
task_pause 3 force
fi
if [[ "$CONFIG_TASK" == "$CALLER_TASK" ]] && [[ "$OPTION_XTRACE" -eq 1 ]]; then
log_notice "Debug mode enabled - heavy debugging in sight, may the \"Force\" be with you!"
task_pause 3 force
fi
if [[ "$OPTION_XTRACE" -eq 1 ]]; then
set -o verbose
set -o xtrace
fi
# ---
local FUNC_STOP="$($CMD_DATE "+%s%1N")"
compute_time "$FUNC_INIT" "$FUNC_STOP"
log_time "$DISPLAY_STAT_TIME_1ST - Check global options"
log_trace "<-- Check global options"
reset_trap
}
# ---
function check_config_value {
local TEMP_NAME="${1:-undefined}"
local TEMP_MODE="${2:-undefined}"
local TEMP_INIT="${3:-}"
local TEMP_SIZE="${4:-}"
if [[ -n "$TEMP_SIZE" ]]; then
local TEMP_MIN="${TEMP_SIZE%-*}"
local TEMP_MAX="${TEMP_SIZE#*-}"
fi
# ---
if [[ "$TEMP_NAME" == "undefined" ]] ||
( [[ "$TEMP_MODE" != "yes_no" ]] && [[ "$TEMP_MODE" != "string" ]] && [[ "$TEMP_MODE" != "integer" ]] && [[ "$TEMP_MODE" != "ctid_list" ]] &&
[[ "$TEMP_MODE" != "ssh_path" ]] && [[ "$TEMP_MODE" != "ssh_options" ]] &&
[[ "$TEMP_MODE" != "rsync_options" ]] ); then
exit_critical "Invalid check_config_value call (name: \"$TEMP_NAME\" - mode: \"$TEMP_MODE\" - init: \"$TEMP_INIT\" - size: \"$TEMP_SIZE\") - please send a bug report"
return
fi
# ---
if [[ -z "${!TEMP_NAME+set}" ]]; then
local TEMP_DATA="$TEMP_INIT"
log_trace "$TEMP_NAME not defined - using default setting (\"$TEMP_DATA\")"
else
local TEMP_DATA="${!TEMP_NAME}"
log_debug "$TEMP_NAME is defined - using *custom* setting (\"$TEMP_DATA\")"
fi
# ---
if [[ "$TEMP_MODE" == "yes_no" ]]; then
if [[ "$TEMP_DATA" != "yes" ]] && [[ "$TEMP_DATA" != "no" ]]; then
exit_error "Invalid $TEMP_NAME (\"$TEMP_DATA\") - must be set to \"yes\" or \"no\" - please check configuration file"
return
fi
readonly $TEMP_NAME="$TEMP_DATA"
export $TEMP_NAME
return 0
fi
# ---
if [[ "$TEMP_MODE" == "string" ]]; then
readonly $TEMP_NAME="$TEMP_DATA"
export $TEMP_NAME
return 0
fi
# ---
if [[ "$TEMP_MODE" == "integer" ]]; then
if [[ "$TEMP_INIT" == "auto" ]]; then
if [[ "$TEMP_DATA" != "auto" ]] && [[ "$TEMP_DATA" == *[!0-9]* ]]; then
exit_error "Invalid $TEMP_NAME (\"$TEMP_DATA\") - must be set to \"auto\" or a positive integer, \"0\" to disable functionnality - please check configuration file"
return
fi
elif [[ -n "$TEMP_SIZE" ]]; then
if [[ "$TEMP_DATA" == *[!0-9]* ]] || [[ "$TEMP_DATA" -gt "$TEMP_MAX" ]]; then
exit_error "Invalid $TEMP_NAME (\"$TEMP_DATA\") - must be a positive integer from \"$TEMP_MIN\" to \"$TEMP_MAX\", \"0\" to disable functionnality - please check configuration file"
return
fi
else
if [[ "$TEMP_DATA" == *[!0-9]* ]]; then
exit_error "Invalid $TEMP_NAME (\"$TEMP_DATA\") - must be a positive integer, \"0\" to disable functionnality - please check configuration file"
return
fi
fi
readonly $TEMP_NAME="$TEMP_DATA"
export $TEMP_NAME
return 0
fi
# ---
if [[ "$TEMP_MODE" == "ctid_list" ]]; then
if [[ -n "$TEMP_DATA" ]] && [[ "$TEMP_DATA" != "all" ]]; then
local TEMP_CTID=""
for TEMP_CTID in $TEMP_DATA; do
if [[ "$TEMP_CTID" == *[!0-9]* ]] && [[ "${#TEMP_CTID}" -ne 36 ]] && [[ "$TEMP_CTID" == *[!0-9a-f\-]* ]]; then
exit_error "Invalid $TEMP_NAME (\"$TEMP_DATA\") - must contain a list of CTID or \"all\" - please check configuration file"
return
fi
done
fi
readonly $TEMP_NAME="$TEMP_DATA"
export $TEMP_NAME
return 0
fi
# ---
if [[ "$TEMP_MODE" == "ssh_path" ]]; then
local TEMP_SSH_MODE="${TEMP_NAME%%_*}"
local TEMP_SSH_PATH="$TEMP_DATA"
local TEMP_SSH_HOST="${TEMP_SSH_PATH#*@}"
local TEMP_SSH_HOST="${TEMP_SSH_HOST%%:*}"
if [[ -z "$TEMP_SSH_HOST" ]] || [[ "$TEMP_SSH_HOST" == "$TEMP_SSH_PATH" ]]; then
exit_error "Invalid $TEMP_NAME (\"$TEMP_SSH_PATH\") - expecting \"root at host.domain.tld:/absolute/path/to/backups\" - please check configuration file"
return
fi
if [[ "$TEMP_SSH_PATH" != *"root@"* ]]; then
exit_error "Invalid $TEMP_NAME (\"$TEMP_SSH_PATH\") - root user is mandatory - please check configuration file"
return
fi
# --- Extract IPV4 or IPV6
local TEMP_IPV4="$($CMD_ECHO "$TEMP_SSH_HOST" | $CMD_GREP -P '^(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)$')"
local TEMP_IPV6="$($CMD_ECHO "$TEMP_SSH_HOST" | $CMD_GREP -P '^((([0-9A-Fa-f]{1,4}:){7}[0-9A-Fa-f]{1,4})|(([0-9A-Fa-f]{1,4}:){6}:[0-9A-Fa-f]{1,4})|(([0-9A-Fa-f]{1,4}:){5}:([0-9A-Fa-f]{1,4}:)?[0-9A-Fa-f]{1,4})|(([0-9A-Fa-f]{1,4}:){4}:([0-9A-Fa-f]{1,4}:){0,2}[0-9A-Fa-f]{1,4})|(([0-9A-Fa-f]{1,4}:){3}:([0-9A-Fa-f]{1,4}:){0,3}[0-9A-Fa-f]{1,4})|(([0-9A-Fa-f]{1,4}:){2}:([0-9A-Fa-f]{1,4}:){0,4}[0-9A-Fa-f]{1,4})|(([0-9A-Fa-f]{1,4}:){6}((b((25[0-5])|(1d{2})|(2[0-4]d)|(d{1,2}))b).){3}(b((25[0-5])|(1d{2})|(2[0-4]d)|(d{1,2}))b))|(([0-9A-Fa-f]{1,4}:){0,5}:((b((25[0-5])|(1d{2})|(2[0-4]d)|(d{1,2}))b).){3}(b((25[0-5])|(1d{2})|(2[0-4]d)|(d{1,2}))b))|(::([0-9A-Fa-f]{1,4}:){0,5}((b((25[0-5])|(1d{2})|(2[0-4]d)|(d{1,2}))b).){3}(b((25[0-5])|(1d{2})|(2[0-4]d)|(d{1,2}))b))|([0-9A-Fa-f]{1,4}::([0-9A-Fa-f]{1,4}:){0,5}[0-9A-Fa-f]{1,4})|(::([0-9A-Fa-f]{1,4}:){0,6}[0-9A-Fa-f]{1,4})|(([0-9A-Fa-f]{1,4}:){1,7}:))$')"
if [[ -n "$TEMP_IPV4" ]]; then
local TEMP_SSH_ADDR="$TEMP_IPV4"
elif [[ -n "$TEMP_IPV6" ]]; then
local TEMP_SSH_ADDR="$TEMP_IPV6"
else
if [[ "$TEMP_SSH_HOST" == "localhost" ]]; then
local TEMP_SSH_ADDR="127.0.0.1"
else
local TEMP_SSH_ADDR="$($CMD_DIG +short A "$TEMP_SSH_HOST" 2>> "$LOG_FILE" | $CMD_HEAD -n 1)"
if [[ -z "$TEMP_SSH_ADDR" ]]; then
local TEMP_SSH_ADDR="$($CMD_DIG +short AAAA "$TEMP_SSH_HOST" 2>> "$LOG_FILE" | $CMD_HEAD -n 1)"
if [[ -z "$TEMP_SSH_ADDR" ]]; then
exit_error "Invalid $TEMP_NAME (\"$TEMP_SSH_HOST\") - IP address not found - please check configuration file"
return
fi
fi
fi
# --- Rewrite
local TEMP_SSH_PATH="${TEMP_SSH_PATH/$TEMP_SSH_HOST/$TEMP_SSH_ADDR}"
local TEMP_SSH_HOST="$TEMP_SSH_ADDR"
log_debug "$TEMP_NAME rewrited to (\"$TEMP_SSH_PATH\")"
fi
# --- Extract SSH path
local TEMP_DIR_PATH="${TEMP_SSH_PATH#*:}"
if [[ -z "$TEMP_DIR_PATH" ]] || [[ "$TEMP_DIR_PATH" == "$TEMP_SSH_PATH" ]]; then
exit_error "Invalid $TEMP_NAME (\"$TEMP_SSH_PATH\") - expecting \"root at host.domain.tld:/absolute/path/to/backups\" - please check configuration file"
return
fi
if [[ ! "$TEMP_DIR_PATH" =~ ^/ ]]; then
exit_error "Invalid $TEMP_NAME (\"$TEMP_DIR_PATH\") - must be an absolute path, starting with a slash - please check configuration file"
return
fi
# ---
local SSH_NAME=""
for SSH_NAME in TEMP_SSH_ADDR TEMP_SSH_HOST TEMP_SSH_PATH TEMP_DIR_PATH; do
local TEMP_VAR="${TEMP_SSH_MODE}_${SSH_NAME#TEMP_}"
readonly $TEMP_VAR="${!SSH_NAME}"
export $TEMP_VAR
done
return 0
fi
# ---
if [[ "$TEMP_MODE" == "ssh_options" ]]; then
if [[ "$TEMP_DATA" != *"ssh"* ]]; then
exit_error "Invalid $TEMP_NAME (\"$TEMP_DATA\") - missing \"ssh\" command - please check configuration file"
return
fi
# --- Rewrite
local TEMP_DATA="$TEMP_DATA -o SendEnv=LC_ALL"
# Note: ALL (check_config_value: SSH mux => insert PID in ControlPath)
if [[ "$CONFIG_TASK" == "$CALLER_TASK" ]]; then
local TEMP_DATA="${TEMP_DATA/${CALLER_TASK}_/${CALLER_TASK}_${BATCH_TASK}_${$}_}"
elif [[ -n "$OPTION_CONFIG" ]] || [[ ! -f "$CONFIG_PATH/etc/${CALLER_TASK}_${BACKUP_CTID}.conf" ]]; then
local TEMP_DATA="${TEMP_DATA/${CALLER_TASK}_/${CALLER_TASK}_${CONFIG_TASK}_${PPID}_}"
else
local TEMP_DATA="${TEMP_DATA/${CALLER_TASK}_/${CALLER_TASK}_${CONFIG_TASK}_${BACKUP_CTID}_}"
fi
log_debug "$TEMP_NAME rewrited to (\"$TEMP_DATA\")"
# Note: mutable
declare -g $TEMP_NAME="$TEMP_DATA"
return 0
fi
# ---
if [[ "$TEMP_MODE" == "rsync_options" ]]; then
if [[ -z "${!TEMP_NAME+set}" ]]; then
# Note: PCS6/VZ7 (check_config_value: [MASTER|REMOTE]_RSYNC_PARAMETERS => disable xattrs and acls support)
if [[ "$CONFIG_TYPE" == "Virtuozzo" ]]; then
local TEMP_DATA=""
log_info "$CONFIG_TYPE ${CONFIG_XVER:-Legacy} detected, disabling extended attributes and ACL support ($TEMP_NAME=\"$TEMP_DATA\")"
else
log_trace "$TEMP_NAME not defined - using default setting (\"$TEMP_DATA\")"
fi
else
log_debug "$TEMP_NAME is defined - using *custom* setting (\"$TEMP_DATA\")"
fi
if [[ -n "$TEMP_DATA" ]]; then
local TEMP_OPTION=""
for TEMP_OPTION in $TEMP_DATA; do
if [[ "$TEMP_OPTION" != "--acls" ]] && [[ "$TEMP_OPTION" != "--xattrs" ]]; then
log_notice "Custom rsync parameter detected (\"$TEMP_OPTION\") - dangerous (let's hope you know what you're doing)"
break
fi
done
fi
readonly $TEMP_NAME="$TEMP_DATA"
export $TEMP_NAME
return 0
fi
}
# ---
function load_configuration {
set_trap "Unable to load configuration"
log_trace "--> Load configuration"
# --- Batch vs task
if [[ "$CONFIG_TASK" == "$CALLER_TASK" ]]; then
local TASK_CTID="$BATCH_CTID"
else
local TASK_CTID="$BACKUP_CTID"
fi
if [[ -n "$OPTION_CONFIG" ]]; then
readonly CONFIG_FILE="$OPTION_CONFIG"
elif [[ -f "$CONFIG_PATH/etc/${CALLER_TASK}_${TASK_CTID}.conf" ]]; then
readonly CONFIG_FILE="$CONFIG_PATH/etc/${CALLER_TASK}_${TASK_CTID}.conf"
elif [[ -f "$CONFIG_PATH/etc/$CALLER_TASK.conf" ]]; then
readonly CONFIG_FILE="$CONFIG_PATH/etc/$CALLER_TASK.conf"
else
readonly CONFIG_FILE="$CONFIG_PATH/etc/$CALLER_TASK.conf"
fi
# --- Cache
if [[ "$CONFIG_TASK" != "$CALLER_TASK" ]] && [[ "${GLOBAL_CONFIG:-}" == "$CONFIG_FILE" ]]; then
log_trace "<-- Load configuration"
reset_trap
return 0
fi
# --- Load configuration file
local FUNC_INIT="$($CMD_DATE "+%s%1N")"
if [[ -n "$OPTION_CONFIG" ]]; then
log_info2 "Loading configuration file (mode: \"inline config\" - name: \"${CONFIG_FILE##*/}\")"
elif [[ -f "$CONFIG_PATH/etc/${CALLER_TASK}_${TASK_CTID}.conf" ]]; then
log_info2 "Loading configuration file (mode: \"custom config\" - name: \"${CONFIG_FILE##*/}\")"
elif [[ -f "$CONFIG_PATH/etc/$CALLER_TASK.conf" ]]; then
log_info2 "Loading configuration file (mode: \"global config\" - name: \"${CONFIG_FILE##*/}\")"
else
log_info2 "Missing configuration file (mode: \"ignore config\" - name: \"$CALLER_TASK.conf.sample\")"
fi
# ---
if [[ -f "$CONFIG_FILE" ]]; then
local TEMP_IFS="$IFS"
# Note: mutable
IFS=$'\n'
local TEMP_LIST="$($CMD_GREP -v -e "^\s*$" -e "^\s*#" "$CONFIG_FILE" 2>> "$LOG_FILE")"
local TEMP_CONF=""
for TEMP_CONF in $TEMP_LIST; do
local TEMP_NAME="${TEMP_CONF%%=*}"
local TEMP_NAME="${TEMP_NAME// /}"
local TEMP_DATA="${TEMP_CONF#*=}"
local TEMP_DATA="${TEMP_DATA// /}"
local TEMP_DATA="${TEMP_DATA#\"}"
local TEMP_DATA="${TEMP_DATA%\"}"
if [[ "$TEMP_NAME" == *[!0-9A-Z_]* ]]; then
exit_error "Invalid configuration option (file: \"$CONFIG_FILE\" - name: \"$TEMP_NAME\" - data: \"$TEMP_DATA\") - aborting"
return
fi
if [[ -n "$TEMP_DATA" ]] && [[ ! "$TEMP_DATA" =~ ^[\-\.\ 0-9A-Za-z_=:@/%]+$ ]]; then
log_notice "Invalid configuration syntax (file: \"$CONFIG_FILE\" - name: \"$TEMP_NAME\" - data: \"$TEMP_DATA\") - ignoring (please, do not panic)"
continue
fi
done
# Note: mutable
IFS="$TEMP_IFS"
# --- Load file
source "$CONFIG_FILE"
fi
# ---
local FUNC_STOP="$($CMD_DATE "+%s%1N")"
compute_time "$FUNC_INIT" "$FUNC_STOP"
log_time "$DISPLAY_STAT_TIME_1ST - Load configuration"
log_trace "<-- Load configuration"
reset_trap
}
# ---
function check_configuration {
set_trap "Unable to check configuration"
log_trace "--> Check configuration"
# ---
local TEMP_STRG="${1:-undefined}"
if [[ "$TEMP_STRG" != "master" ]] && [[ "$TEMP_STRG" != "remote" ]]; then
exit_critical "Invalid check_configuration call (strg: \"$TEMP_STRG\") - please send a bug report"
return
fi
# --- Cache
if [[ "$CONFIG_TASK" != "$CALLER_TASK" ]] && [[ "${GLOBAL_CONFIG:-}" == "$CONFIG_FILE" ]]; then
local TEMP_CONF=""
# GENERAL
for TEMP_CONF in EMAIL_ADDRESS EMAIL_ADDRESS_DEBUG EMAIL_ADDRESS_ERROR MD5_SOURCE_HASH MD5_MASTER_HASH MD5_REMOTE_HASH MD5_TARGET_HASH; do
readonly $TEMP_CONF="${!TEMP_CONF}"
done
# MISC
for TEMP_CONF in KICK_VZCTL_PROCESS CONTAINERS_FORCE_COLD_BACKUP CONTAINERS_FORCE_HOLD_BACKUP CONTAINERS_FORCE_LIVE_BACKUP; do
readonly $TEMP_CONF="${!TEMP_CONF}"
done
# SERVER
for TEMP_CONF in SERVER_VZCONF_DIR SERVER_TMP_DIR SERVER_TMPFS_SIZE SERVER_SNAPSHOT_SIZE; do
readonly $TEMP_CONF="${!TEMP_CONF}"
done
# MASTER
if [[ "$CONFIG_TASK" == "backup" ]] || [[ "$CONFIG_TASK" == "delete" ]] || [[ "$CONFIG_TASK" == "destroy" ]] || [[ "$CONFIG_TASK" == "download" ]] || [[ "$CONFIG_TASK" == "upload" ]] || [[ "$CONFIG_TASK" == "restore" ]]; then
for TEMP_CONF in MASTER_SSH_PATH MASTER_SSH_OPTIONS MASTER_RSYNC_PARAMETERS MASTER_RSYNC_COMPRESSION_LEVEL MASTER_RSYNC_TIMEOUT MASTER_RSYNC_LOOKUP_SPEED MASTER_RSYNC_BACKUP_SPEED MASTER_RSYNC_DELETE_SPEED MASTER_RSYNC_DOWNLOAD_SPEED MASTER_RSYNC_UPLOAD_SPEED; do
readonly $TEMP_CONF="${!TEMP_CONF}"
done
fi
# REMOTE
if [[ "$CONFIG_TASK" == "download" ]] || [[ "$CONFIG_TASK" == "upload" ]] ||
( [[ "$CONFIG_TASK" == "delete" ]] && [[ "$TEMP_STRG" == "remote" ]] ) ||
( [[ "$CONFIG_TASK" == "destroy" ]] && [[ "$TEMP_STRG" == "remote" ]] ); then
for TEMP_CONF in REMOTE_SSH_PATH REMOTE_SSH_OPTIONS REMOTE_RSYNC_PARAMETERS REMOTE_RSYNC_COMPRESSION_LEVEL REMOTE_RSYNC_TIMEOUT REMOTE_RSYNC_LOOKUP_SPEED REMOTE_RSYNC_BACKUP_SPEED REMOTE_RSYNC_DELETE_SPEED REMOTE_RSYNC_DOWNLOAD_SPEED REMOTE_RSYNC_UPLOAD_SPEED; do
readonly $TEMP_CONF="${!TEMP_CONF}"
done
fi
# BACKUP
if [[ "$CONFIG_TASK" == "backup" ]]; then
for TEMP_CONF in BACKUP_SKIP_HOSTS BACKUP_SKIP_CONTAINERS BACKUP_SKIP_STOPPED_CONTAINERS BACKUP_SKIP_RUNNING_CONTAINERS; do
readonly $TEMP_CONF="${!TEMP_CONF}"
done
fi
# DELETE
if [[ "$CONFIG_TASK" == "delete" ]]; then
for TEMP_CONF in DELETE_SKIP_HOSTS DELETE_SKIP_CONTAINERS DELETE_SKIP_STOPPED_CONTAINERS DELETE_SKIP_RUNNING_CONTAINERS; do
readonly $TEMP_CONF="${!TEMP_CONF}"
done
fi
# DESTROY
if [[ "$CONFIG_TASK" == "destroy" ]]; then
for TEMP_CONF in DESTROY_SKIP_HOSTS DESTROY_SKIP_CONTAINERS DESTROY_SKIP_STOPPED_CONTAINERS DESTROY_SKIP_RUNNING_CONTAINERS; do
readonly $TEMP_CONF="${!TEMP_CONF}"
done
fi
# DOWNLOAD
if [[ "$CONFIG_TASK" == "download" ]]; then
for TEMP_CONF in DOWNLOAD_SKIP_HOSTS DOWNLOAD_SKIP_CONTAINERS DOWNLOAD_SKIP_STOPPED_CONTAINERS DOWNLOAD_SKIP_RUNNING_CONTAINERS; do
readonly $TEMP_CONF="${!TEMP_CONF}"
done
fi
# UPLOAD
if [[ "$CONFIG_TASK" == "upload" ]]; then
for TEMP_CONF in UPLOAD_SKIP_HOSTS UPLOAD_SKIP_CONTAINERS UPLOAD_SKIP_STOPPED_CONTAINERS UPLOAD_SKIP_RUNNING_CONTAINERS; do
readonly $TEMP_CONF="${!TEMP_CONF}"
done
fi
# RESTORE
if [[ "$CONFIG_TASK" == "restore" ]]; then
for TEMP_CONF in RESTORE_SKIP_HOSTS RESTORE_SKIP_CONTAINERS RESTORE_SKIP_STOPPED_CONTAINERS RESTORE_SKIP_RUNNING_CONTAINERS; do
readonly $TEMP_CONF="${!TEMP_CONF}"
done
fi
# ---
readonly CMD_MAGIC_BZIP2="$CMD_MAGIC_BZIP2"
readonly CMD_SOURCE_NOCACHE="$CMD_SOURCE_NOCACHE"
# ---
log_trace "<-- Check configuration"
reset_trap
return 0
fi
# ---
local FUNC_INIT="$($CMD_DATE "+%s%1N")"
###########
# GENERAL #
###########
check_config_value "EMAIL_ADDRESS" "string"
# ---
check_config_value "EMAIL_ADDRESS_DEBUG" "string"
if [[ -z "$EMAIL_ADDRESS" ]] && [[ -z "$EMAIL_ADDRESS_DEBUG" ]]; then
log_info2 "EMAIL_ADDRESS_DEBUG is empty - debug notifications are disabled"
fi
# ---
check_config_value "EMAIL_ADDRESS_ERROR" "string"
if [[ -z "$EMAIL_ADDRESS" ]] && [[ -z "$EMAIL_ADDRESS_ERROR" ]]; then
log_info2 "EMAIL_ADDRESS_ERROR is empty - error notifications are disabled"
fi
# ---
if [[ -z "$EMAIL_ADDRESS" ]] && [[ -z "$EMAIL_ADDRESS_DEBUG" ]] && [[ -z "$EMAIL_ADDRESS_ERROR" ]]; then
log_info2 "EMAIL_ADDRESS, EMAIL_ADDRESS_DEBUG and EMAIL_ADDRESS_ERROR are undefined - all notifications are disabled"
fi
# ---
check_config_value "MD5_SOURCE_HASH" "yes_no" "yes"
check_config_value "MD5_MASTER_HASH" "yes_no" "yes"
check_config_value "MD5_REMOTE_HASH" "yes_no" "yes"
check_config_value "MD5_TARGET_HASH" "yes_no" "yes"
########
# MISC #
########
check_config_value "KICK_VZCTL_PROCESS" "yes_no" "no"
check_config_value "CONTAINERS_FORCE_COLD_BACKUP" "ctid_list"
check_config_value "CONTAINERS_FORCE_HOLD_BACKUP" "ctid_list"
check_config_value "CONTAINERS_FORCE_LIVE_BACKUP" "ctid_list"
##########
# SERVER #
##########
check_config_value "SERVER_VZCONF_DIR" "string" "/etc/vz/conf"
if [[ ! -d "$SERVER_VZCONF_DIR/" ]] && ( [[ "$BATCH_TASK" == "backup" ]] || ( [[ "$BATCH_TASK" == "restore" ]] && [[ "$CALLER_ARGS" != "$BATCH_TASK ${BACKUP_CTID:-$BATCH_CTID} list"* ]] ) ); then
exit_error "Invalid SERVER_VZCONF_DIR (\"$SERVER_VZCONF_DIR\") - directory not found - please check configuration file"
return
fi
# ---
check_config_value "SERVER_TMP_DIR" "string" "/var/tmp"
if [[ ! -d "$SERVER_TMP_DIR/" ]]; then
exit_error "Invalid SERVER_TMP_DIR (\"$SERVER_TMP_DIR\") - directory not found - please check configuration file"
return
fi
# ---
check_config_value "SERVER_TMPFS_SIZE" "integer" "auto"
check_config_value "SERVER_SNAPSHOT_SIZE" "integer" "auto"
##########
# MASTER #
##########
if [[ "$BATCH_TASK" == "backup" ]] || [[ "$BATCH_TASK" == "delete" ]] || [[ "$BATCH_TASK" == "destroy" ]] || [[ "$BATCH_TASK" == "download" ]] || [[ "$BATCH_TASK" == "upload" ]] || [[ "$BATCH_TASK" == "restore" ]]; then
check_config_value "MASTER_SSH_PATH" "ssh_path" "root at localhost:/home/backup/$CALLER_TASK"
check_config_value "MASTER_SSH_OPTIONS" "ssh_options" "ssh -p 22 -c aes128-ctr -o Compression=no -o ConnectTimeout=60 -o ControlMaster=auto -o ControlPath=/root/${CALLER_TASK}_%r@%h:%p -o ControlPersist=3600 -o ForwardX11=no -o PasswordAuthentication=no -o PreferredAuthentications=publickey -o StrictHostKeyChecking=ask"
check_config_value "MASTER_RSYNC_PARAMETERS" "rsync_options" "--acls --xattrs"
check_config_value "MASTER_RSYNC_COMPRESSION_LEVEL" "integer" "1" "1-9"
check_config_value "MASTER_RSYNC_TIMEOUT" "integer" "3600"
check_config_value "MASTER_RSYNC_LOOKUP_SPEED" "integer" "4096"
check_config_value "MASTER_RSYNC_BACKUP_SPEED" "integer" "4096"
check_config_value "MASTER_RSYNC_DELETE_SPEED" "integer" "4096"
check_config_value "MASTER_RSYNC_DOWNLOAD_SPEED" "integer" "12500"
check_config_value "MASTER_RSYNC_UPLOAD_SPEED" "integer" "12500"
fi
##########
# REMOTE #
##########
if [[ "$BATCH_TASK" == "download" ]] || [[ "$BATCH_TASK" == "upload" ]] ||
( [[ "$BATCH_TASK" == "delete" ]] && [[ "$TEMP_STRG" == "remote" ]] ) ||
( [[ "$BATCH_TASK" == "destroy" ]] && [[ "$TEMP_STRG" == "remote" ]] ); then
check_config_value "REMOTE_SSH_PATH" "ssh_path" "root at localhost:/home/backup/$CALLER_TASK"
check_config_value "REMOTE_SSH_OPTIONS" "ssh_options" "ssh -p 22 -c aes128-ctr -o Compression=no -o ConnectTimeout=60 -o ControlMaster=auto -o ControlPath=/root/${CALLER_TASK}_%r@%h:%p -o ControlPersist=3600 -o ForwardX11=no -o PasswordAuthentication=no -o PreferredAuthentications=publickey -o StrictHostKeyChecking=ask"
check_config_value "REMOTE_RSYNC_PARAMETERS" "rsync_options" "--acls --xattrs"
check_config_value "REMOTE_RSYNC_COMPRESSION_LEVEL" "integer" "1" "1-9"
check_config_value "REMOTE_RSYNC_TIMEOUT" "integer" "3600"
check_config_value "REMOTE_RSYNC_LOOKUP_SPEED" "integer" "4096"
check_config_value "REMOTE_RSYNC_BACKUP_SPEED" "integer" "4096"
check_config_value "REMOTE_RSYNC_DELETE_SPEED" "integer" "4096"
check_config_value "REMOTE_RSYNC_DOWNLOAD_SPEED" "integer" "12500"
check_config_value "REMOTE_RSYNC_UPLOAD_SPEED" "integer" "12500"
fi
##########
# BACKUP #
##########
if [[ "$BATCH_TASK" == "backup" ]]; then
check_config_value "BACKUP_SKIP_HOSTS" "string" "others"
check_config_value "BACKUP_SKIP_CONTAINERS" "ctid_list"
check_config_value "BACKUP_SKIP_STOPPED_CONTAINERS" "ctid_list"
check_config_value "BACKUP_SKIP_RUNNING_CONTAINERS" "ctid_list"
fi
##########
# DELETE #
##########
if [[ "$BATCH_TASK" == "delete" ]]; then
check_config_value "DELETE_SKIP_HOSTS" "string" "others"
check_config_value "DELETE_SKIP_CONTAINERS" "ctid_list"
check_config_value "DELETE_SKIP_STOPPED_CONTAINERS" "ctid_list"
check_config_value "DELETE_SKIP_RUNNING_CONTAINERS" "ctid_list"
fi
###########
# DESTROY #
###########
if [[ "$BATCH_TASK" == "destroy" ]]; then
check_config_value "DESTROY_SKIP_HOSTS" "string" "others"
check_config_value "DESTROY_SKIP_CONTAINERS" "ctid_list"
check_config_value "DESTROY_SKIP_STOPPED_CONTAINERS" "ctid_list"
check_config_value "DESTROY_SKIP_RUNNING_CONTAINERS" "ctid_list"
fi
############
# DOWNLOAD #
############
if [[ "$BATCH_TASK" == "download" ]]; then
check_config_value "DOWNLOAD_SKIP_HOSTS" "string" ""
check_config_value "DOWNLOAD_SKIP_CONTAINERS" "ctid_list"
check_config_value "DOWNLOAD_SKIP_STOPPED_CONTAINERS" "ctid_list"
check_config_value "DOWNLOAD_SKIP_RUNNING_CONTAINERS" "ctid_list"
fi
##########
# UPLOAD #
##########
if [[ "$BATCH_TASK" == "upload" ]]; then
check_config_value "UPLOAD_SKIP_HOSTS" "string" "others"
check_config_value "UPLOAD_SKIP_CONTAINERS" "ctid_list"
check_config_value "UPLOAD_SKIP_STOPPED_CONTAINERS" "ctid_list"
check_config_value "UPLOAD_SKIP_RUNNING_CONTAINERS" "ctid_list"
fi
###########
# RESTORE #
###########
if [[ "$BATCH_TASK" == "restore" ]]; then
check_config_value "RESTORE_SKIP_HOSTS" "string" ""
check_config_value "RESTORE_SKIP_CONTAINERS" "ctid_list"
check_config_value "RESTORE_SKIP_STOPPED_CONTAINERS" "ctid_list"
check_config_value "RESTORE_SKIP_RUNNING_CONTAINERS" "ctid_list"
fi
# --- Set options
if [[ "$BATCH_TASK" == "backup" ]] || [[ "$BATCH_TASK" == "delete" ]] || [[ "$BATCH_TASK" == "destroy" ]] || [[ "$BATCH_TASK" == "download" ]] || [[ "$BATCH_TASK" == "upload" ]] || [[ "$BATCH_TASK" == "restore" ]]; then
if [[ "$OPTION_VERBOSE" -gt 2 ]]; then
MASTER_SSH_OPTIONS="$MASTER_SSH_OPTIONS -v"
elif [[ "$OPTION_VERBOSE" -gt 1 ]]; then
MASTER_SSH_OPTIONS="$MASTER_SSH_OPTIONS"
elif [[ "$OPTION_VERBOSE" -gt 0 ]]; then
MASTER_SSH_OPTIONS="$MASTER_SSH_OPTIONS"
else
MASTER_SSH_OPTIONS="$MASTER_SSH_OPTIONS -q"
fi
# readonly MASTER_SSH_OPTIONS="$MASTER_SSH_OPTIONS"
# export MASTER_SSH_OPTIONS
fi
# ---
if [[ "$BATCH_TASK" == "download" ]] || [[ "$BATCH_TASK" == "upload" ]] ||
( [[ "$BATCH_TASK" == "delete" ]] && [[ "$TEMP_STRG" == "remote" ]] ) ||
( [[ "$BATCH_TASK" == "destroy" ]] && [[ "$TEMP_STRG" == "remote" ]] ); then
if [[ "$OPTION_VERBOSE" -gt 2 ]]; then
REMOTE_SSH_OPTIONS="$REMOTE_SSH_OPTIONS -v"
elif [[ "$OPTION_VERBOSE" -gt 1 ]]; then
REMOTE_SSH_OPTIONS="$REMOTE_SSH_OPTIONS"
elif [[ "$OPTION_VERBOSE" -gt 0 ]]; then
REMOTE_SSH_OPTIONS="$REMOTE_SSH_OPTIONS"
else
REMOTE_SSH_OPTIONS="$REMOTE_SSH_OPTIONS -q"
fi
# readonly REMOTE_SSH_OPTIONS="$REMOTE_SSH_OPTIONS"
# export REMOTE_SSH_OPTIONS
fi
# ---
# Note: ALL (check_configuration: OPTION_OPTIMIZE & 4 => downgrade to CMD_BZIP2)
if [[ -n "$CMD_PBZIP2" ]] && [[ "$(( OPTION_OPTIMIZE & 4 ))" -eq 4 ]]; then
log_info2 "Optimized mode ( 4) - using \"$CMD_BZIP2\" (mono-core) instead of \"$CMD_PBZIP2\" (multi-core)"
readonly CMD_MAGIC_BZIP2="$CMD_BZIP2"
elif [[ -n "$CMD_PBZIP2" ]]; then
readonly CMD_MAGIC_BZIP2="$CMD_PBZIP2"
else
readonly CMD_MAGIC_BZIP2="$CMD_BZIP2"
fi
export CMD_MAGIC_BZIP2
# --- Check nocache (source host)
# Note: ALL (check_configuration: OPTION_OPTIMIZE & 16 => enable CMD_SOURCE_NOCACHE)
if [[ "$(( OPTION_OPTIMIZE & 16 ))" -eq 16 ]]; then
local TEMP_CMD_SOURCE_NOCACHE="$($CMD_NOCACHE $CMD_ECHO -n "OK" 2>&1)"
if [[ "$TEMP_CMD_SOURCE_NOCACHE" == "OK" ]]; then
log_info2 "Optimized mode (16) - \"$CMD_NOCACHE\" detected - bypassing kernel page cache (source host)"
CMD_SOURCE_NOCACHE="$CMD_NOCACHE"
else
log_info2 "Unable to detect \"$CMD_NOCACHE\" on source host (\"$TEMP_CMD_SOURCE_NOCACHE\") - ignoring"
fi
fi
readonly CMD_SOURCE_NOCACHE="${CMD_SOURCE_NOCACHE:-}"
export CMD_SOURCE_NOCACHE
# ---
local FUNC_STOP="$($CMD_DATE "+%s%1N")"
compute_time "$FUNC_INIT" "$FUNC_STOP"
log_time "$DISPLAY_STAT_TIME_1ST - Check configuration"
log_trace "<-- Check configuration"
reset_trap
}
# ---
function check_runtime_configuration {
set_trap "Unable to check runtime configuration (${1:-undefined} storage)"
log_trace "--> Check runtime configuration (${1:-undefined} storage)"
# ---
local TEMP_STRG="${1:-undefined}"
if [[ "$TEMP_STRG" != "master" ]] && [[ "$TEMP_STRG" != "remote" ]]; then
exit_critical "Invalid check_runtime_configuration call (strg: \"$TEMP_STRG\") - please send a bug report"
return
fi
# --- Cache
if [[ "$CONFIG_TASK" != "$CALLER_TASK" ]] && [[ "${GLOBAL_CONFIG:-}" == "$CONFIG_FILE" ]]; then
# --- Check directory (master host)
if [[ "$TEMP_STRG" == "master" ]]; then
# readonly MASTER_SSH_OPTIONS="$MASTER_SSH_OPTIONS"
readonly MASTER_BUSYBOX="$MASTER_BUSYBOX"
# readonly ARGS_CHOWNMOD="$ARGS_CHOWNMOD"
# readonly ARGS_CPMKMVRM="$ARGS_CPMKMVRM"
readonly MASTER_IONICE="$MASTER_IONICE"
readonly MASTER_NICE="$MASTER_NICE"
# readonly MASTER_STORAGE_EXIST="$MASTER_STORAGE_EXIST"
readonly CMD_MASTER_NOCACHE="$CMD_MASTER_NOCACHE"
# ---
local TEMP_MASTER_STORAGE="$($MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "if [ -d \"$MASTER_DIR_PATH/$BACKUP_CTID/\" ]; then echo -n \"OK\"; else echo -n \"KO\"; fi" 2>> "$LOG_FILE")"
if [[ "$TEMP_MASTER_STORAGE" == "OK" ]]; then
readonly MASTER_STORAGE_EXIST="yes"
# export MASTER_STORAGE_EXIST
elif [[ "$TEMP_MASTER_STORAGE" == "KO" ]]; then
readonly MASTER_STORAGE_EXIST="no"
# export MASTER_STORAGE_EXIST
# Note: ALL (check_runtime_configuration: SSH failure or bug)
else
exit_critical "Unable to detect storage on $TEMP_STRG host (\"$TEMP_MASTER_STORAGE\") - please send a bug report"
return
fi
fi
# --- Check directory (remote host)
if [[ "$TEMP_STRG" == "remote" ]]; then
# readonly REMOTE_SSH_OPTIONS="$REMOTE_SSH_OPTIONS"
readonly REMOTE_BUSYBOX="$REMOTE_BUSYBOX"
# readonly ARGS_CHOWNMOD="$ARGS_CHOWNMOD"
# readonly ARGS_CPMKMVRM="$ARGS_CPMKMVRM"
readonly REMOTE_IONICE="$REMOTE_IONICE"
readonly REMOTE_NICE="$REMOTE_NICE"
# readonly REMOTE_STORAGE_EXIST="$REMOTE_STORAGE_EXIST"
readonly CMD_REMOTE_NOCACHE="$CMD_REMOTE_NOCACHE"
# ---
local TEMP_REMOTE_STORAGE="$($MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "$REMOTE_SSH_OPTIONS \"$REMOTE_SSH_HOST\" \"if [ -d \\\"$REMOTE_DIR_PATH/$BACKUP_CTID/\\\" ]; then echo -n \\\"OK\\\"; else echo -n \\\"KO\\\"; fi\"" 2>> "$LOG_FILE")"
if [[ "$TEMP_REMOTE_STORAGE" == "OK" ]]; then
readonly REMOTE_STORAGE_EXIST="yes"
# export REMOTE_STORAGE_EXIST
elif [[ "$TEMP_REMOTE_STORAGE" == "KO" ]]; then
readonly REMOTE_STORAGE_EXIST="no"
# export REMOTE_STORAGE_EXIST
# Note: ALL (check_runtime_configuration: SSH failure or bug)
else
exit_critical "Unable to detect storage on $TEMP_STRG host (\"$TEMP_REMOTE_STORAGE\") - please send a bug report"
return
fi
fi
# ---
log_trace "<-- Check runtime configuration"
reset_trap
return 0
fi
# ---
local FUNC_INIT="$($CMD_DATE "+%s%1N")"
if [[ "$TEMP_STRG" == "master" ]]; then
set_speed "check_runtime_configuration" "nice"
MASTER_SSH_OPTIONS="$CMD_IONICE -t -c "$IONICE_CLASS" -n "$IONICE_LEVEL" $CMD_NICE -n "$NICE_LEVEL" $MASTER_SSH_OPTIONS"
# --- Check SSH (source host)
# Note: ALL (check_runtime_configuration: MASTER_SSH_OPTIONS => STDERR is needed)
local TEMP_ARGS="$($MASTER_SSH_OPTIONS -V 2>&1)"
if [[ "${TEMP_ARGS,,}" == *"controlpersist"* ]]; then
log_info2 "SSH \"ControlPersist\" option not supported (\"$TEMP_ARGS\") - auto-ajusting parameters"
MASTER_SSH_OPTIONS="$($CMD_ECHO $MASTER_SSH_OPTIONS | $CMD_SED 's/\-o ControlPersist=[0-9]*//i')"
fi
readonly MASTER_SSH_OPTIONS="$MASTER_SSH_OPTIONS"
export MASTER_SSH_OPTIONS
# --- Check SSH connection (master host)
set +o errexit; trap - ERR
# Note: ALL (check_runtime_configuration: no output => "2>&1 | $CMD_TEE -a $LOG_FILE" will hang)
$MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "$CMD_ECHO -n" 2>> "$LOG_FILE"
local ERROR_CODE="$?"
set_trap "Unable to check runtime configuration"
# --- Check error code
if [[ "$ERROR_CODE" -ne 0 ]]; then
exit_error "SSH error: connection failure (host: \"$MASTER_SSH_HOST\" - error: \"$ERROR_CODE\") - please check SSH private/public keys"
return
fi
# --- Check BusyBox (master host)
local TEMP_MASTER_BUSYBOX="$($MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "$CMD_READLINK -f -n \"\$($CMD_WHICH $CMD_LS)\"" 2>> "$LOG_FILE")"
if [[ "${TEMP_MASTER_BUSYBOX##*/}" == "busybox" ]]; then
log_info2 "\"BusyBox\" detected on $TEMP_STRG host (path: \"$TEMP_MASTER_BUSYBOX\") - auto-ajusting parameters"
readonly MASTER_BUSYBOX="yes"
export MASTER_BUSYBOX
readonly ARGS_CHOWNMOD=""
export ARGS_CHOWNMOD
readonly ARGS_CPMKMVRM=""
export ARGS_CPMKMVRM
elif [[ "${TEMP_MASTER_BUSYBOX##*/}" == "ls" ]]; then
readonly MASTER_BUSYBOX="no"
export MASTER_BUSYBOX
if [[ -z "${REMOTE_SSH_HOST:-}" ]]; then
readonly ARGS_CHOWNMOD="$ARGS_CHOWNMOD"
export ARGS_CHOWNMOD
readonly ARGS_CPMKMVRM="$ARGS_CPMKMVRM"
export ARGS_CPMKMVRM
fi
# Note: ALL (check_runtime_configuration: SSH failure or bug)
else
exit_critical "Unable to detect BusyBox on $TEMP_STRG host (\"$TEMP_MASTER_BUSYBOX\") - please send a bug report"
return
fi
# --- Check ionice (master host)
local TEMP_MASTER_IONICE="$($MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "$CMD_IONICE $CMD_ECHO -n \"OK\"" 2>&1)"
if [[ "$TEMP_MASTER_IONICE" == "OK" ]]; then
readonly MASTER_IONICE="yes"
export MASTER_IONICE
# Note: ALL (check_runtime_configuration: SSH failure or bug)
else
log_info2 "Unable to detect \"$CMD_IONICE\" on $TEMP_STRG host (\"$TEMP_MASTER_IONICE\") - ignoring"
readonly MASTER_IONICE="no"
export MASTER_IONICE
fi
# --- Check nice (master host)
local TEMP_MASTER_NICE="$($MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "$CMD_NICE $CMD_ECHO -n \"OK\"" 2>&1)"
if [[ "$TEMP_MASTER_NICE" == "OK" ]]; then
readonly MASTER_NICE="yes"
export MASTER_NICE
# Note: ALL (check_runtime_configuration: SSH failure or bug)
else
log_info2 "Unable to detect \"$CMD_NICE\" on $TEMP_STRG host (\"$TEMP_MASTER_NICE\") - ignoring"
readonly MASTER_NICE="no"
export MASTER_NICE
fi
# --- Check rsync (master host)
if [[ "$MASTER_BUSYBOX" == "yes" ]]; then
local TEMP_MASTER_RSYNC="$($MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "$CMD_READLINK -f -n \"\$($CMD_WHICH $CMD_RSYNC)\"" 2>> "$LOG_FILE")"
else
local TEMP_MASTER_RSYNC="$($MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "$CMD_READLINK -e -n \"\$($CMD_WHICH $CMD_RSYNC)\"" 2>> "$LOG_FILE")"
fi
# Note: ALL (check_runtime_configuration: SSH failure or bug)
if [[ "${TEMP_MASTER_RSYNC##*/}" != "$CMD_RSYNC" ]]; then
exit_error "Unable to detect $CMD_RSYNC on $TEMP_STRG host (\"$TEMP_MASTER_RSYNC\") - aborting"
return
fi
# --- Check directory (master host)
if [[ -n "${BACKUP_CTID:-}" ]]; then
local TEMP_MASTER_STORAGE="$($MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "if [ -d \"$MASTER_DIR_PATH/$BACKUP_CTID/\" ]; then echo -n \"OK\"; else echo -n \"KO\"; fi" 2>> "$LOG_FILE")"
if [[ "$TEMP_MASTER_STORAGE" == "OK" ]]; then
readonly MASTER_STORAGE_EXIST="yes"
export MASTER_STORAGE_EXIST
elif [[ "$TEMP_MASTER_STORAGE" == "KO" ]]; then
readonly MASTER_STORAGE_EXIST="no"
export MASTER_STORAGE_EXIST
# Note: ALL (check_runtime_configuration: SSH failure or bug)
else
exit_critical "Unable to detect storage on $TEMP_STRG host (\"$TEMP_MASTER_STORAGE\") - please send a bug report"
return
fi
fi
# --- Check nocache (master host)
# Note: ALL (check_runtime_configuration: OPTION_OPTIMIZE & 16 => enable CMD_MASTER_NOCACHE)
if [[ "$(( OPTION_OPTIMIZE & 16 ))" -eq 16 ]]; then
local TEMP_CMD_MASTER_NOCACHE="$($MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "$CMD_NOCACHE $CMD_ECHO -n \"OK\"" 2>&1)"
if [[ "$TEMP_CMD_MASTER_NOCACHE" == "OK" ]]; then
log_info2 "Optimized mode (16) - \"$CMD_NOCACHE\" detected - bypassing kernel page cache ($TEMP_STRG host)"
CMD_MASTER_NOCACHE="$CMD_NOCACHE"
# Note: ALL (check_runtime_configuration: SSH failure or bug)
else
log_info2 "Unable to detect \"$CMD_NOCACHE\" on $TEMP_STRG host (\"$TEMP_CMD_MASTER_NOCACHE\") - ignoring"
fi
fi
readonly CMD_MASTER_NOCACHE="${CMD_MASTER_NOCACHE:-}"
export CMD_MASTER_NOCACHE
fi
# ---
if [[ "$TEMP_STRG" == "remote" ]]; then
set_speed "check_runtime_configuration" "nice"
if [[ "$MASTER_IONICE" == "yes" ]] && [[ "$IONICE_CLASS" -eq "3" ]] && [[ "$MASTER_NICE" == "yes" ]]; then
REMOTE_SSH_OPTIONS="$CMD_IONICE -c "$IONICE_CLASS" $CMD_NICE -n "$NICE_LEVEL" $REMOTE_SSH_OPTIONS"
elif [[ "$MASTER_IONICE" == "yes" ]] && [[ "$IONICE_CLASS" -eq "3" ]]; then
REMOTE_SSH_OPTIONS="$CMD_IONICE -c "$IONICE_CLASS" $REMOTE_SSH_OPTIONS"
elif [[ "$MASTER_IONICE" == "yes" ]] && [[ "$MASTER_NICE" == "yes" ]]; then
REMOTE_SSH_OPTIONS="$CMD_IONICE -c "$IONICE_CLASS" -n "$IONICE_LEVEL" $CMD_NICE -n "$NICE_LEVEL" $REMOTE_SSH_OPTIONS"
elif [[ "$MASTER_IONICE" == "yes" ]]; then
REMOTE_SSH_OPTIONS="$CMD_IONICE -c "$IONICE_CLASS" -n "$IONICE_LEVEL" $REMOTE_SSH_OPTIONS"
elif [[ "$MASTER_NICE" == "yes" ]]; then
REMOTE_SSH_OPTIONS="$CMD_NICE -n "$NICE_LEVEL" $REMOTE_SSH_OPTIONS"
# else
# REMOTE_SSH_OPTIONS="$REMOTE_SSH_OPTIONS"
fi
# --- Check SSH (master host)
# Note: ALL (check_runtime_configuration: REMOTE_SSH_OPTIONS => STDERR is needed)
local TEMP_ARGS="$($MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "$REMOTE_SSH_OPTIONS -V" 2>&1)"
if [[ "${TEMP_ARGS,,}" == *"controlpersist"* ]]; then
log_info2 "SSH \"ControlPersist\" option not supported (\"$TEMP_ARGS\") - auto-ajusting parameters"
REMOTE_SSH_OPTIONS="$($CMD_ECHO $REMOTE_SSH_OPTIONS | $CMD_SED 's/\-o ControlPersist=[0-9]*//i')"
fi
readonly REMOTE_SSH_OPTIONS="$REMOTE_SSH_OPTIONS"
export REMOTE_SSH_OPTIONS
# --- Check SSH connection (remote host)
set +o errexit; trap - ERR
# Note: ALL (check_runtime_configuration: no output => "2>&1 | $CMD_TEE -a $LOG_FILE" will hang)
$MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "$REMOTE_SSH_OPTIONS \"$REMOTE_SSH_HOST\" \"$CMD_ECHO -n\"" 2>> "$LOG_FILE"
local ERROR_CODE="$?"
set_trap "Unable to check runtime configuration"
# --- Check error code
if [[ "$ERROR_CODE" -ne 0 ]]; then
exit_error "SSH error: connection failure (host: \"$MASTER_SSH_HOST\" -> \"$REMOTE_SSH_HOST\" - error: \"$ERROR_CODE\") - please check SSH private/public keys"
return
fi
# --- Check BusyBox (remote host)
local TEMP_REMOTE_BUSYBOX="$($MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "$REMOTE_SSH_OPTIONS \"$REMOTE_SSH_HOST\" \"$CMD_READLINK -f -n \\\"\\\$($CMD_WHICH $CMD_LS)\\\"\"" 2>> "$LOG_FILE")"
if [[ "${TEMP_REMOTE_BUSYBOX##*/}" == "busybox" ]]; then
log_info2 "\"BusyBox\" detected on $TEMP_STRG host (path: \"$TEMP_REMOTE_BUSYBOX\") - auto-ajusting parameters"
readonly REMOTE_BUSYBOX="yes"
export REMOTE_BUSYBOX
if [[ "$MASTER_BUSYBOX" != "yes" ]]; then
readonly ARGS_CHOWNMOD=""
export ARGS_CHOWNMOD
readonly ARGS_CPMKMVRM=""
export ARGS_CPMKMVRM
fi
elif [[ "${TEMP_REMOTE_BUSYBOX##*/}" == "ls" ]]; then
readonly REMOTE_BUSYBOX="no"
export REMOTE_BUSYBOX
if [[ "$MASTER_BUSYBOX" != "yes" ]]; then
readonly ARGS_CHOWNMOD="$ARGS_CHOWNMOD"
export ARGS_CHOWNMOD
readonly ARGS_CPMKMVRM="$ARGS_CPMKMVRM"
export ARGS_CPMKMVRM
fi
# Note: ALL (check_runtime_configuration: SSH failure or bug)
else
exit_critical "Unable to detect BusyBox on $TEMP_STRG host (\"$TEMP_REMOTE_BUSYBOX\") - please send a bug report"
return
fi
# --- Check ionice (remote host)
local TEMP_REMOTE_IONICE="$($MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "$REMOTE_SSH_OPTIONS \"$REMOTE_SSH_HOST\" \"$CMD_IONICE $CMD_ECHO -n \\\"OK\\\"\"" 2>&1)"
if [[ "$TEMP_REMOTE_IONICE" == "OK" ]]; then
readonly REMOTE_IONICE="yes"
export REMOTE_IONICE
# Note: ALL (check_runtime_configuration: SSH failure or bug)
else
log_info2 "Unable to detect \"$CMD_IONICE\" on $TEMP_STRG host (\"$TEMP_REMOTE_IONICE\") - ignoring"
readonly REMOTE_IONICE="no"
export REMOTE_IONICE
fi
# --- Check nice (remote host)
local TEMP_REMOTE_NICE="$($MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "$REMOTE_SSH_OPTIONS \"$REMOTE_SSH_HOST\" \"$CMD_NICE $CMD_ECHO -n \\\"OK\\\"\"" 2>&1)"
if [[ "$TEMP_REMOTE_NICE" == "OK" ]]; then
readonly REMOTE_NICE="yes"
export REMOTE_NICE
# Note: ALL (check_runtime_configuration: SSH failure or bug)
else
log_info2 "Unable to detect \"$CMD_NICE\" on $TEMP_STRG host (\"$TEMP_REMOTE_NICE\") - ignoring"
readonly REMOTE_NICE="no"
export REMOTE_NICE
fi
# --- Check rsync (remote host)
if [[ "$REMOTE_BUSYBOX" == "yes" ]]; then
local TEMP_REMOTE_RSYNC="$($MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "$REMOTE_SSH_OPTIONS \"$REMOTE_SSH_HOST\" \"$CMD_READLINK -f -n \\\"\\\$($CMD_WHICH $CMD_RSYNC)\\\"\"" 2>> "$LOG_FILE")"
else
local TEMP_REMOTE_RSYNC="$($MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "$REMOTE_SSH_OPTIONS \"$REMOTE_SSH_HOST\" \"$CMD_READLINK -e -n \\\"\\\$($CMD_WHICH $CMD_RSYNC)\\\"\"" 2>> "$LOG_FILE")"
fi
# Note: ALL (check_runtime_configuration: SSH failure or bug)
if [[ "${TEMP_REMOTE_RSYNC##*/}" != "$CMD_RSYNC" ]]; then
exit_error "Unable to detect $CMD_RSYNC on $TEMP_STRG host (\"$TEMP_REMOTE_RSYNC\") - aborting"
return
fi
# --- Check directory (remote host)
if [[ -n "${BACKUP_CTID:-}" ]]; then
local TEMP_REMOTE_STORAGE="$($MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "$REMOTE_SSH_OPTIONS \"$REMOTE_SSH_HOST\" \"if [ -d \\\"$REMOTE_DIR_PATH/$BACKUP_CTID/\\\" ]; then echo -n \\\"OK\\\"; else echo -n \\\"KO\\\"; fi\"" 2>> "$LOG_FILE")"
if [[ "$TEMP_REMOTE_STORAGE" == "OK" ]]; then
readonly REMOTE_STORAGE_EXIST="yes"
export REMOTE_STORAGE_EXIST
elif [[ "$TEMP_REMOTE_STORAGE" == "KO" ]]; then
readonly REMOTE_STORAGE_EXIST="no"
export REMOTE_STORAGE_EXIST
# Note: ALL (check_runtime_configuration: SSH failure or bug)
else
exit_critical "Unable to detect storage on $TEMP_STRG host (\"$TEMP_REMOTE_STORAGE\") - please send a bug report"
return
fi
fi
# --- Check nocache (remote host)
# Note: ALL (check_runtime_configuration: OPTION_OPTIMIZE & 16 => enable CMD_REMOTE_NOCACHE)
if [[ "$(( OPTION_OPTIMIZE & 16 ))" -eq 16 ]]; then
local TEMP_CMD_REMOTE_NOCACHE="$($MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "$REMOTE_SSH_OPTIONS \"$REMOTE_SSH_HOST\" \"$CMD_NOCACHE $CMD_ECHO -n \\\"OK\\\"\"" 2>&1)"
if [[ "$TEMP_CMD_REMOTE_NOCACHE" == "OK" ]]; then
log_info2 "Optimized mode (16) - \"$CMD_NOCACHE\" detected - bypassing kernel page cache ($TEMP_STRG host)"
CMD_REMOTE_NOCACHE="$CMD_NOCACHE"
# Note: ALL (check_runtime_configuration: SSH failure or bug)
else
log_info2 "Unable to detect \"$CMD_NOCACHE\" on $TEMP_STRG host (\"$TEMP_CMD_REMOTE_NOCACHE\") - ignoring"
fi
fi
readonly CMD_REMOTE_NOCACHE="${CMD_REMOTE_NOCACHE:-}"
export CMD_REMOTE_NOCACHE
fi
# ---
local FUNC_STOP="$($CMD_DATE "+%s%1N")"
compute_time "$FUNC_INIT" "$FUNC_STOP"
log_time "$DISPLAY_STAT_TIME_1ST - Check runtime configuration ($TEMP_STRG storage)"
log_trace "<-- Check runtime configuration"
reset_trap
}
# ---
function check_container_status {
set_trap "Unable to check container status"
log_trace "--> Check container status"
local FUNC_INIT="$($CMD_DATE "+%s%1N")"
# --- task_restore
if [[ -n "${TARGET_CTID:-}" ]] && [[ "$TARGET_CTID" != "$BACKUP_CTID" ]]; then
log_debug "Switching backup/target CTID (backup: $BACKUP_CTID - target: $TARGET_CTID)"
local TASK_CTID="$TARGET_CTID"
else
local TASK_CTID="$BACKUP_CTID"
fi
# ---
# Note: PCS6/VZ7 (check_container_status: CTID => VEID)
if [[ "$CONFIG_TYPE" == "Virtuozzo" ]]; then
# Note: mutable
CT_STATUS="VEID $TASK_CTID deleted unmounted down"
else
# Note: mutable
CT_STATUS="CTID $TASK_CTID deleted unmounted down"
fi
# Note: mutable
CT_STATUS_EXIST="deleted"
CT_STATUS_MOUNT="unmounted"
CT_STATUS_RUNNING="down"
CT_STATUS_SUSPENDED=""
# --- Check proc entry
# Note: VZ7 (check_container_status: "/proc/cpt" not available => ignore)
if [[ ! -f "/proc/cpt" ]] && ( [[ "$CONFIG_TYPE" != "Virtuozzo" ]] || [[ "$CONFIG_XVER" -ne 7 ]] ); then
if [[ "$BATCH_TASK" == "backup" ]] || ( [[ "$BATCH_TASK" == "restore" ]] && [[ "$CALLER_ARGS" != "$BATCH_TASK ${BACKUP_CTID:-$BATCH_CTID} list"* ]] && [[ "$CALLER_ARGS" != "$BATCH_TASK ${BATCH_CTID:-$BACKUP_CTID} list"* ]] ); then
exit_critical "Proc entry not found (path: \"/proc/cpt\") - please send a bug report"
return
fi
log_debug "Proc entry not found (path: \"/proc/cpt\") - skipping"
fi
# --- Get container status
if [[ -n "$CMD_VZCTL" ]] && [[ -f "$SERVER_VZCONF_DIR/$TASK_CTID.conf" ]]; then
# Note: mutable
CT_STATUS="$($CMD_VZCTL status "$TASK_CTID" 2>> "$LOG_FILE" || true)"
fi
if [[ -z "$CT_STATUS" ]]; then
if [[ "$CONFIG_TYPE" != "Virtuozzo" ]] && [[ "${#TASK_CTID}" -eq 36 ]] && [[ "$TASK_CTID" == *[0-9a-f\-]* ]]; then
# Note: mutable
CT_STATUS="CTID $TASK_CTID deleted unmounted down"
else
exit_critical "Unable to check container status (ctid: $TASK_CTID - status: \"$CT_STATUS\") - please send a bug report"
return
fi
fi
# ---
declare -a TEMP_ARRAY="()"
IFS=" " read -a TEMP_ARRAY <<< "$CT_STATUS"
local TEMP_INDEX="2"
local TEMP_COUNT="${#TEMP_ARRAY[@]}"
local TEMP_NAME=""
for TEMP_NAME in CT_STATUS_EXIST CT_STATUS_MOUNT CT_STATUS_RUNNING CT_STATUS_SUSPENDED; do
if [[ "$TEMP_INDEX" -lt "$TEMP_COUNT" ]]; then
# Note: mutable
declare -g $TEMP_NAME="${TEMP_ARRAY[$TEMP_INDEX]}"
else
# Note: mutable
declare -g $TEMP_NAME=""
fi
local TEMP_INDEX="$(( TEMP_INDEX + 1 ))"
done
# ---
if [[ "$CT_STATUS_EXIST" != "deleted" ]] && [[ "$CT_STATUS_EXIST" != "exist" ]]; then
exit_critical "Invalid container status (ctid: $TASK_CTID - status: \"$CT_STATUS\") - please send a bug report"
return
fi
if [[ "$CT_STATUS_MOUNT" != "unmounted" ]] && [[ "$CT_STATUS_MOUNT" != "mounted" ]]; then
exit_critical "Invalid container status (ctid: $TASK_CTID - status: \"$CT_STATUS\") - please send a bug report"
return
fi
if [[ "$CT_STATUS_RUNNING" != "down" ]] && [[ "$CT_STATUS_RUNNING" != "running" ]]; then
exit_critical "Invalid container status (ctid: $TASK_CTID - status: \"$CT_STATUS\") - please send a bug report"
return
fi
if [[ "$CT_STATUS_SUSPENDED" != "" ]] && [[ "$CT_STATUS_SUSPENDED" != "suspended" ]]; then
exit_critical "Invalid container status (ctid: $TASK_CTID - status: \"$CT_STATUS\") - please send a bug report"
return
fi
# ---
# Note: OPENVZ/PCS6/PVE2/PVE3 (check_container_status: "/proc/cpt" => suspended containers)
if [[ -f "/proc/cpt" ]] && [[ "$CT_STATUS_SUSPENDED" == "" ]]; then
local TEMP_LIST="$($CMD_CAT "/proc/cpt" | $CMD_GREP -v "^Ctx" | $CMD_CUT -d " " -f "3" -s)"
# local TEMP_LIST="${TEMP_LIST//$'\r'/ }"
local TEMP_LIST="${TEMP_LIST//$'\n'/ }"
# local TEMP_LIST="${TEMP_LIST// / }"
local TEMP_CTID=""
for TEMP_CTID in $TEMP_LIST; do
if [[ "$CONFIG_TYPE" == "Virtuozzo" ]] && [[ "$TEMP_CTID" == *[!0-9]* ]] && [[ "${#TEMP_CTID}" -ne 36 ]] && [[ "$TEMP_CTID" == *[!0-9a-f\-]* ]]; then
# Note: PCS6 (check_container_status: fallback)
log_warning "Unexpected proc entry (ctid: \"$TEMP_CTID\" - list: \"$TEMP_LIST\") - please send a bug report if this happens again"
continue
elif [[ "$CONFIG_TYPE" != "Virtuozzo" ]] && [[ "$TEMP_CTID" == *[!0-9]* ]]; then
# Note: OPENVZ/PVE2/PVE3 (check_container_status: fallback)
log_warning "Unexpected proc entry (ctid: \"$TEMP_CTID\" - list: \"$TEMP_LIST\") - please send a bug report if this happens again"
continue
fi
if [[ "$TEMP_CTID" != "$TASK_CTID" ]]; then
# Note: OPENVZ/PCS6/PVE2/PVE3 (check_container_status: OPTION_WAIT => synchronized backups)
log_info2 "Another container is suspended (ctid: \"$TEMP_CTID\") - use \"$CMD_VZCTL chkpnt $TEMP_CTID --resume\" to resume it manually"
continue
fi
# Note: mutable
CT_STATUS_SUSPENDED="suspended"
CT_STATUS="$CT_STATUS suspended"
done
fi
# ---
local FUNC_STOP="$($CMD_DATE "+%s%1N")"
compute_time "$FUNC_INIT" "$FUNC_STOP"
log_time "$DISPLAY_STAT_TIME_1ST - Check container status (\"$CT_STATUS\")"
log_trace "<-- Check container status"
reset_trap
}
# ---
function lock_container {
set_trap "Unable to lock container"
log_trace "--> Lock container"
local FUNC_INIT="$($CMD_DATE "+%s%1N")"
# --- Alredy locked
if [[ -n "${SOURCE_LOCK:-}" ]]; then
exit_critical "Unable to lock container (SOURCE_LOCK is already set: \"$SOURCE_LOCK\") - please send a bug report"
return
fi
# --- Set lock directory
# FIXME (low): hardcoded (TEMP_DIR - /var/tmp)
local TEMP_DIR="/var/tmp/${CALLER_TASK}_${BACKUP_CTID}.lock"
# --- Create lock directory
if [[ "$OPTION_DRYRUN" -eq 1 ]]; then
local TASK_INIT="$($CMD_DATE "+%s%1N")"
dry_run "$CMD_MKDIR $ARGS_CPMKMVRM \"$TEMP_DIR/\" && \
$CMD_CHOWN $ARGS_CHOWNMOD \"root:root\" \"$TEMP_DIR/\" && \
$CMD_CHMOD $ARGS_CHOWNMOD \"0750\" \"$TEMP_DIR/\""
local TASK_STOP="$($CMD_DATE "+%s%1N")"
else
local TASK_INIT="$($CMD_DATE "+%s%1N")"
set +o errexit; trap - ERR
$CMD_MKDIR $ARGS_CPMKMVRM "$TEMP_DIR/" 2>&1 | $CMD_TEE -a "$LOG_FILE" && \
$CMD_CHOWN $ARGS_CHOWNMOD "root:root" "$TEMP_DIR/" 2>&1 | $CMD_TEE -a "$LOG_FILE" && \
$CMD_CHMOD $ARGS_CHOWNMOD "0750" "$TEMP_DIR/" 2>&1 | $CMD_TEE -a "$LOG_FILE"
local ERROR_CODE="$?"
set_trap "Unable to lock container"
local TASK_STOP="$($CMD_DATE "+%s%1N")"
# --- Check error code
if [[ "$ERROR_CODE" -ne 0 ]]; then
log_info "Unable to lock container (path: \"$TEMP_DIR\" - error: \"$ERROR_CODE\") - checking lock time"
local LOCK_TIME="$($CMD_STAT -c "%Y" "$TEMP_DIR/" 2>> "$LOG_FILE")"
if [[ "$LOCK_TIME" == *[!0-9]* ]]; then
exit_critical "Unable to check container lock (path: \"$TEMP_DIR\" - time: \"$LOCK_TIME\") - please send a bug report"
return
fi
# ---
local UNIX_TIME="$($CMD_DATE "+%s")"
local DIFF_TIME="$($CMD_PRINTF "%2d" "$(( ( UNIX_TIME - LOCK_TIME ) / 3600 ))")h$($CMD_PRINTF "%2d" "$(( ( ( UNIX_TIME - LOCK_TIME ) % 3600 ) / 60 ))")m$($CMD_PRINTF "%2d" "$(( ( ( UNIX_TIME - LOCK_TIME ) % 3600 ) % 60 ))")s"
local DIFF_TIME="${DIFF_TIME# 0h}"
local DIFF_TIME="${DIFF_TIME# 0m}"
local DIFF_TIME="${DIFF_TIME# }"
local DIFF_TIME="${DIFF_TIME// /0}"
# ---
# FIXME (low): hardcoded (LOCK_TIME > 12 hours - expired)
if [[ "$(( LOCK_TIME + 43200 ))" -lt "$UNIX_TIME" ]]; then
log_notice "Container lock has expired (previous lock set $DIFF_TIME ago) - ignoring (please, do not panic)"
elif [[ "$OPTION_UNLOCK" -eq 1 ]]; then
log_warning "Bypassing container lock (previous lock set $DIFF_TIME ago) - dangerous (let's hope you know what you're doing)"
else
exit_error "Unable to lock container (previous lock set $DIFF_TIME ago) - aborting (please, wait until lock is released and try again)"
return
fi
# --- Update lock
set +o errexit; trap - ERR
$CMD_TOUCH "$TEMP_DIR/" 2>&1 | $CMD_TEE -a "$LOG_FILE"
local ERROR_CODE_2="$?"
set_trap "Unable to lock container"
local TASK_STOP="$($CMD_DATE "+%s%1N")"
# --- Check error code 2
if [[ "$ERROR_CODE_2" -ne 0 ]]; then
exit_critical "Unable to update container lock (path: \"$TEMP_DIR\" - error: \"$ERROR_CODE_2\") - please send a bug report"
return
fi
fi
fi
# Note: mutable
SOURCE_LOCK="$CONFIG_TASK"
# ---
local FUNC_STOP="$($CMD_DATE "+%s%1N")"
compute_time "$TASK_INIT" "$TASK_STOP"
log_time "$DISPLAY_STAT_TIME_1ST - Lock container (path: \"$TEMP_DIR\")"
log_trace "<-- Lock container"
reset_trap
}
# ---
function unlock_container {
set_trap "Unable to unlock container"
log_trace "--> Unlock container"
local FUNC_INIT="$($CMD_DATE "+%s%1N")"
# --- Not locked
if [[ -z "${SOURCE_LOCK:-}" ]]; then
log_info2 "Unable to unlock container (SOURCE_LOCK is unset) - ignoring"
log_trace "<-- Unlock container"
reset_trap
return 0
fi
# --- Set lock directory
# FIXME (low): hardcoded (TEMP_DIR - /var/tmp)
local TEMP_DIR="/var/tmp/${CALLER_TASK}_${BACKUP_CTID}.lock"
# --- Delete lock directory
if [[ "$OPTION_DRYRUN" -eq 1 ]]; then
local TASK_INIT="$($CMD_DATE "+%s%1N")"
dry_run "$CMD_RMDIR $ARGS_CPMKMVRM \"$TEMP_DIR/\""
local TASK_STOP="$($CMD_DATE "+%s%1N")"
else
local TASK_INIT="$($CMD_DATE "+%s%1N")"
set +o errexit; trap - ERR
$CMD_RMDIR $ARGS_CPMKMVRM "$TEMP_DIR/" 2>&1 | $CMD_TEE -a "$LOG_FILE"
local ERROR_CODE="$?"
set_trap "Unable to unlock container"
local TASK_STOP="$($CMD_DATE "+%s%1N")"
# --- Check error code
if [[ "$ERROR_CODE" -ne 0 ]]; then
# Note: ALL (unlock_container: fallback)
log_warning "Unable to unlock container (path: \"$TEMP_DIR\" - error: \"$ERROR_CODE\") - please send a bug report if this happens again"
fi
fi
# Note: mutable
unset SOURCE_LOCK
# ---
local FUNC_STOP="$($CMD_DATE "+%s%1N")"
compute_time "$TASK_INIT" "$TASK_STOP"
log_time "$DISPLAY_STAT_TIME_1ST - Unlock container (path: \"$TEMP_DIR\")"
log_trace "<-- Unlock container"
reset_trap
}
# ---
function lock_storage {
set_trap "Unable to lock ${1:-undefined} storage"
log_trace "--> Lock ${1:-undefined} storage"
local FUNC_INIT="$($CMD_DATE "+%s%1N")"
# ---
local TEMP_STRG="${1:-undefined}"
if [[ "$TEMP_STRG" != "master" ]] && [[ "$TEMP_STRG" != "remote" ]]; then
exit_critical "Invalid lock_storage call (strg: \"$TEMP_STRG\") - please send a bug report"
return
fi
# --- Already locked
if [[ "$TEMP_STRG" == "master" ]] && [[ -n "${MASTER_LOCK:-}" ]]; then
exit_critical "Unable to lock $TEMP_STRG storage (MASTER_LOCK is already set: \"$MASTER_LOCK\") - please send a bug report"
return
elif [[ "$TEMP_STRG" == "remote" ]] && [[ -n "${REMOTE_LOCK:-}" ]]; then
exit_critical "Unable to lock $TEMP_STRG storage (REMOTE_LOCK is already set: \"$REMOTE_LOCK\") - please send a bug report"
return
fi
# --- Set lock directory
if [[ "$TEMP_STRG" == "master" ]]; then
local TEMP_DIR="$MASTER_DIR_PATH/$BACKUP_CTID/lock"
elif [[ "$TEMP_STRG" == "remote" ]]; then
local TEMP_DIR="$REMOTE_DIR_PATH/$BACKUP_CTID/lock"
fi
# --- Create lock directory
if [[ "$OPTION_DRYRUN" -eq 1 ]]; then
local TASK_INIT="$($CMD_DATE "+%s%1N")"
if [[ "$TEMP_STRG" == "master" ]]; then
dry_run "$MASTER_SSH_OPTIONS \"$MASTER_SSH_HOST\" \" \
$CMD_MKDIR $ARGS_CPMKMVRM \\\"$TEMP_DIR/\\\" && \
$CMD_CHOWN $ARGS_CHOWNMOD \\\"root:root\\\" \\\"$TEMP_DIR\\\" && \
$CMD_CHMOD $ARGS_CHOWNMOD \\\"0750\\\" \\\"$TEMP_DIR\\\" \
\""
elif [[ "$TEMP_STRG" == "remote" ]]; then
dry_run "$MASTER_SSH_OPTIONS \"$MASTER_SSH_HOST\" \"$REMOTE_SSH_OPTIONS \\\"$REMOTE_SSH_HOST\\\" \\\" \
$CMD_MKDIR $ARGS_CPMKMVRM \\\\\"$TEMP_DIR/\\\\\" && \
$CMD_CHOWN $ARGS_CHOWNMOD \\\\\"root:root\\\\\" \\\\\"$TEMP_DIR\\\\\" && \
$CMD_CHMOD $ARGS_CHOWNMOD \\\\\"0750\\\\\" \\\\\"$TEMP_DIR\\\\\" \
\\\"\""
fi
local TASK_STOP="$($CMD_DATE "+%s%1N")"
# ---
if [[ "$TEMP_STRG" == "master" ]]; then
local TEMP_USAGE="$($MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "$CMD_DF -P \"$MASTER_DIR_PATH/$BACKUP_CTID/\"" 2>> "$LOG_FILE" | $CMD_GREP "[0-9]*%" | $CMD_TR -s " " | $CMD_CUT -d " " -f "5" -s | $CMD_TR -d "%")"
elif [[ "$TEMP_STRG" == "remote" ]]; then
local TEMP_USAGE="$($MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "$REMOTE_SSH_OPTIONS \"$REMOTE_SSH_HOST\" \"$CMD_DF -P \\\"$REMOTE_DIR_PATH/$BACKUP_CTID/\\\"\"" 2>> "$LOG_FILE" | $CMD_GREP "[0-9]*%" | $CMD_TR -s " " | $CMD_CUT -d " " -f "5" -s | $CMD_TR -d "%")"
fi
# Note: ALL (lock_storage: fallback)
if [[ "$TEMP_USAGE" == *[!0-9]* ]]; then
if [[ "$TEMP_STRG" == "master" ]]; then
log_debug "Unable to check $TEMP_STRG storage usage (path: \"$MASTER_DIR_PATH/$BACKUP_CTID\" - used: \"$TEMP_USAGE\" %) - ignoring (dry-run mode)"
local TEMP_USAGE="$($MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "$CMD_DF -P \"$MASTER_DIR_PATH/\"" 2>> "$LOG_FILE" | $CMD_GREP "[0-9]*%" | $CMD_TR -s " " | $CMD_CUT -d " " -f "5" -s | $CMD_TR -d "%")"
elif [[ "$TEMP_STRG" == "remote" ]]; then
log_debug "Unable to check $TEMP_STRG storage usage (path: \"$REMOTE_DIR_PATH/$BACKUP_CTID\" - used: \"$TEMP_USAGE\" %) - ignoring (dry-run mode)"
local TEMP_USAGE="$($MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "$REMOTE_SSH_OPTIONS \"$REMOTE_SSH_HOST\" \"$CMD_DF -P \\\"$REMOTE_DIR_PATH/\\\"\"" 2>> "$LOG_FILE" | $CMD_GREP "[0-9]*%" | $CMD_TR -s " " | $CMD_CUT -d " " -f "5" -s | $CMD_TR -d "%")"
fi
fi
# Note: ALL (lock_storage: fallback)
if [[ "$TEMP_USAGE" == *[!0-9]* ]]; then
if [[ "$TEMP_STRG" == "master" ]]; then
log_debug "Unable to check $TEMP_STRG storage usage (path: \"$MASTER_DIR_PATH\" - used: \"$TEMP_USAGE\" %) - ignoring (dry-run mode)"
local TEMP_USAGE="$($MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "$CMD_DF -P \"/\"" 2>> "$LOG_FILE" | $CMD_GREP "[0-9]*%" | $CMD_TR -s " " | $CMD_CUT -d " " -f "5" -s | $CMD_TR -d "%")"
elif [[ "$TEMP_STRG" == "remote" ]]; then
log_debug "Unable to check $TEMP_STRG storage usage (path: \"$REMOTE_DIR_PATH\" - used: \"$TEMP_USAGE\" %) - ignoring (dry-run mode)"
local TEMP_USAGE="$($MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "$REMOTE_SSH_OPTIONS \"$REMOTE_SSH_HOST\" \"$CMD_DF -P \\\"/\\\"\"" 2>> "$LOG_FILE" | $CMD_GREP "[0-9]*%" | $CMD_TR -s " " | $CMD_CUT -d " " -f "5" -s | $CMD_TR -d "%")"
fi
fi
# Note: ALL (lock_storage: SSH failure or bug)
if [[ "$TEMP_USAGE" == *[!0-9]* ]]; then
exit_critical "Unable to check $TEMP_STRG storage usage (path: \"/\" - used: \"$TEMP_USAGE\" %) - please send a bug report"
return
fi
if [[ "$TEMP_STRG" == "master" ]]; then
readonly MASTER_STORAGE_USAGE="$TEMP_USAGE"
elif [[ "$TEMP_STRG" == "remote" ]]; then
readonly REMOTE_STORAGE_USAGE="$TEMP_USAGE"
fi
else
local TASK_INIT="$($CMD_DATE "+%s%1N")"
set +o errexit; trap - ERR
if [[ "$TEMP_STRG" == "master" ]]; then
$MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" " \
$CMD_MKDIR $ARGS_CPMKMVRM \"$TEMP_DIR/\" && \
$CMD_CHOWN $ARGS_CHOWNMOD \"root:root\" \"$TEMP_DIR/\" && \
$CMD_CHMOD $ARGS_CHOWNMOD \"0750\" \"$TEMP_DIR/\" \
" 2>&1 | $CMD_TEE -a "$LOG_FILE"
elif [[ "$TEMP_STRG" == "remote" ]]; then
$MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "$REMOTE_SSH_OPTIONS \"$REMOTE_SSH_HOST\" \" \
$CMD_MKDIR $ARGS_CPMKMVRM \\\"$TEMP_DIR/\\\" && \
$CMD_CHOWN $ARGS_CHOWNMOD \\\"root:root\\\" \\\"$TEMP_DIR/\\\" && \
$CMD_CHMOD $ARGS_CHOWNMOD \\\"0750\\\" \\\"$TEMP_DIR/\\\" \
\"" 2>&1 | $CMD_TEE -a "$LOG_FILE"
fi
local ERROR_CODE="$?"
set_trap "Unable to lock $TEMP_STRG storage"
local TASK_STOP="$($CMD_DATE "+%s%1N")"
# --- Check error code
if [[ "$ERROR_CODE" -ne 0 ]]; then
if [[ "$ERROR_CODE" -eq 255 ]]; then
if [[ "$TEMP_STRG" == "master" ]]; then
exit_critical "SSH error: connection failure (host: \"$MASTER_SSH_HOST\" - error: \"$ERROR_CODE\") - aborting"
return
elif [[ "$TEMP_STRG" == "remote" ]]; then
exit_critical "SSH error: connection failure (host: \"$MASTER_SSH_HOST\" -> \"$REMOTE_SSH_HOST\" - error: \"$ERROR_CODE\") - aborting"
return
fi
fi
# ---
log_info "Unable to lock $TEMP_STRG storage (path: \"$TEMP_DIR\" - error: \"$ERROR_CODE\") - checking lock time"
if [[ "$TEMP_STRG" == "master" ]]; then
local LOCK_TIME="$($MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "$CMD_STAT -c \"%Y\" \"$TEMP_DIR/\"" 2>> "$LOG_FILE")"
elif [[ "$TEMP_STRG" == "remote" ]]; then
local LOCK_TIME="$($MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "$REMOTE_SSH_OPTIONS \"$REMOTE_SSH_HOST\" \"$CMD_STAT -c \\\"%Y\\\" \\\"$TEMP_DIR/\\\"\"" 2>> "$LOG_FILE")"
fi
# Note: ALL (lock_storage: SSH failure or bug)
if [[ "$LOCK_TIME" == *[!0-9]* ]]; then
exit_critical "Unable to check $TEMP_STRG storage lock (path: \"$TEMP_DIR\" - time: \"$LOCK_TIME\") - please send a bug report"
return
fi
# ---
local UNIX_TIME="$($CMD_DATE "+%s")"
local DIFF_TIME="$($CMD_PRINTF "%2d" "$(( ( UNIX_TIME - LOCK_TIME ) / 3600 ))")h$($CMD_PRINTF "%2d" "$(( ( ( UNIX_TIME - LOCK_TIME ) % 3600 ) / 60 ))")m$($CMD_PRINTF "%2d" "$(( ( ( UNIX_TIME - LOCK_TIME ) % 3600 ) % 60 ))")s"
local DIFF_TIME="${DIFF_TIME# 0h}"
local DIFF_TIME="${DIFF_TIME# 0m}"
local DIFF_TIME="${DIFF_TIME# }"
local DIFF_TIME="${DIFF_TIME// /0}"
# ---
# FIXME (low): hardcoded (LOCK_TIME > 12 hours - expired)
if [[ "$(( LOCK_TIME + 43200 ))" -lt "$UNIX_TIME" ]]; then
log_notice "${TEMP_STRG^} storage lock has expired (previous lock set $DIFF_TIME ago) - ignoring (please, do not panic)"
elif [[ "$OPTION_UNLOCK" -eq 1 ]]; then
log_warning "Bypassing $TEMP_STRG storage lock (previous lock set $DIFF_TIME ago) - dangerous (let's hope you know what you're doing)"
else
exit_error "Unable to lock $TEMP_STRG storage (previous lock set $DIFF_TIME ago) - aborting (please, wait until lock is released and try again)"
return
fi
# --- Update lock
set +o errexit; trap - ERR
if [[ "$TEMP_STRG" == "master" ]]; then
$MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "$CMD_TOUCH \"$TEMP_DIR/\"" 2>&1 | $CMD_TEE -a "$LOG_FILE"
elif [[ "$TEMP_STRG" == "remote" ]]; then
$MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "$REMOTE_SSH_OPTIONS \"$REMOTE_SSH_HOST\" \"$CMD_TOUCH \\\"$TEMP_DIR/\\\"\"" 2>&1 | $CMD_TEE -a "$LOG_FILE"
fi
local ERROR_CODE_2="$?"
set_trap "Unable to lock $TEMP_STRG storage"
local TASK_STOP="$($CMD_DATE "+%s%1N")"
# --- Check error code 2
if [[ "$ERROR_CODE_2" -ne 0 ]]; then
if [[ "$ERROR_CODE_2" -eq 255 ]]; then
if [[ "$TEMP_STRG" == "master" ]]; then
exit_critical "SSH error: connection failure (host: \"$MASTER_SSH_HOST\" - error: \"$ERROR_CODE_2\") - aborting"
return
elif [[ "$TEMP_STRG" == "remote" ]]; then
exit_critical "SSH error: connection failure (host: \"$MASTER_SSH_HOST\" -> \"$REMOTE_SSH_HOST\" - error: \"$ERROR_CODE_2\") - aborting"
return
fi
fi
exit_error "Unable to update $TEMP_STRG storage lock (path: \"$TEMP_DIR\" - error: \"$ERROR_CODE_2\") - aborting"
return
fi
fi
# --- Check storage capacity
if [[ "$TEMP_STRG" == "master" ]]; then
readonly MASTER_STORAGE_USAGE="$($MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "$CMD_DF -P \"$MASTER_DIR_PATH/$BACKUP_CTID/\"" 2>> "$LOG_FILE" | $CMD_GREP "[0-9]*%" | $CMD_TR -s " " | $CMD_CUT -d " " -f "5" -s | $CMD_TR -d "%")"
# Note: ALL (lock_storage: SSH failure or bug)
if [[ "$MASTER_STORAGE_USAGE" == *[!0-9]* ]]; then
exit_critical "Unable to check $TEMP_STRG storage usage (path: \"$MASTER_DIR_PATH/$BACKUP_CTID\" - usage: \"$MASTER_STORAGE_USAGE\" %) - please send a bug report"
return
fi
elif [[ "$TEMP_STRG" == "remote" ]]; then
readonly REMOTE_STORAGE_USAGE="$($MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "$REMOTE_SSH_OPTIONS \"$REMOTE_SSH_HOST\" \"$CMD_DF -P \\\"$REMOTE_DIR_PATH/$BACKUP_CTID/\\\"\"" 2>> "$LOG_FILE" | $CMD_GREP "[0-9]*%" | $CMD_TR -s " " | $CMD_CUT -d " " -f "5" -s | $CMD_TR -d "%")"
# Note: ALL (lock_storage: SSH failure or bug)
if [[ "$REMOTE_STORAGE_USAGE" == *[!0-9]* ]]; then
exit_critical "Unable to check $TEMP_STRG storage usage (path: \"$REMOTE_DIR_PATH/$BACKUP_CTID\" - usage: \"$REMOTE_STORAGE_USAGE\" %) - please send a bug report"
return
fi
fi
fi
if [[ "$TEMP_STRG" == "master" ]]; then
# Note: mutable
MASTER_LOCK="$CONFIG_TASK"
elif [[ "$TEMP_STRG" == "remote" ]]; then
# Note: mutable
REMOTE_LOCK="$CONFIG_TASK"
fi
# ---
if [[ "$TEMP_STRG" == "master" ]]; then
# FIXME (low): hardcoded (MASTER_STORAGE_USAGE >= 98% - aborting)
if [[ "$MASTER_STORAGE_USAGE" -gt 97 ]]; then
if [[ "$CONFIG_TASK" == "backup" ]] || [[ "$CONFIG_TASK" == "download" ]]; then
exit_error "Master storage usage is overlimit: $MASTER_STORAGE_USAGE% (path: \"$MASTER_DIR_PATH/$BACKUP_CTID\") - aborting"
return
fi
log_notice "Master storage usage is overlimit: $MASTER_STORAGE_USAGE% (path: \"$MASTER_DIR_PATH/$BACKUP_CTID\") - ignoring (please, do not panic)"
# FIXME (low): hardcoded (MASTER_STORAGE_USAGE >= 95% - notice)
elif [[ "$MASTER_STORAGE_USAGE" -gt 94 ]]; then
log_notice "Master storage usage is very high: $MASTER_STORAGE_USAGE% (path: \"$MASTER_DIR_PATH/$BACKUP_CTID\") - ignoring (please, do not panic)"
# FIXME (low): hardcoded (MASTER_STORAGE_USAGE >= 90% - info)
elif [[ "$MASTER_STORAGE_USAGE" -gt 89 ]]; then
log_info "Master storage usage is quite high: $MASTER_STORAGE_USAGE% (path: \"$MASTER_DIR_PATH/$BACKUP_CTID\")"
else
log_info2 "Master storage usage: $MASTER_STORAGE_USAGE% (path: \"$MASTER_DIR_PATH/$BACKUP_CTID\")"
fi
elif [[ "$TEMP_STRG" == "master" ]] && [[ -n "${MASTER_STORAGE_USAGE:-}" ]]; then
log_info2 "Master storage usage: $MASTER_STORAGE_USAGE% (path: \"$MASTER_DIR_PATH/$BACKUP_CTID\")"
fi
# ---
if [[ "$TEMP_STRG" == "remote" ]]; then
# FIXME (low): hardcoded (REMOTE_STORAGE_USAGE >= 98% - aborting)
if [[ "$REMOTE_STORAGE_USAGE" -gt 97 ]]; then
if [[ "$CONFIG_TASK" == "upload" ]]; then
exit_error "Remote storage usage is overlimit: $REMOTE_STORAGE_USAGE% (path: \"$REMOTE_DIR_PATH/$BACKUP_CTID\") - aborting"
return
fi
log_notice "Remote storage usage is overlimit: $REMOTE_STORAGE_USAGE% (path: \"$REMOTE_DIR_PATH/$BACKUP_CTID\") - ignoring (please, do not panic)"
# FIXME (low): hardcoded (REMOTE_STORAGE_USAGE >= 95% - notice)
elif [[ "$REMOTE_STORAGE_USAGE" -gt 94 ]]; then
log_notice "Remote storage usage is very high: $REMOTE_STORAGE_USAGE% (path: \"$REMOTE_DIR_PATH/$BACKUP_CTID\") - ignoring (please, do not panic)"
# FIXME (low): hardcoded (REMOTE_STORAGE_USAGE >= 90% - info)
elif [[ "$REMOTE_STORAGE_USAGE" -gt 89 ]]; then
log_info "Remote storage usage is quite high: $REMOTE_STORAGE_USAGE% (path: \"$REMOTE_DIR_PATH/$BACKUP_CTID\")"
else
log_info2 "Remote storage usage: $REMOTE_STORAGE_USAGE% (path: \"$REMOTE_DIR_PATH/$BACKUP_CTID\")"
fi
elif [[ "$TEMP_STRG" == "remote" ]] && [[ -n "${REMOTE_STORAGE_USAGE:-}" ]]; then
log_info2 "Remote storage usage: $REMOTE_STORAGE_USAGE% (path: \"$REMOTE_DIR_PATH/$BACKUP_CTID\")"
fi
# ---
local FUNC_STOP="$($CMD_DATE "+%s%1N")"
compute_time "$TASK_INIT" "$TASK_STOP"
log_time "$DISPLAY_STAT_TIME_1ST - Lock $TEMP_STRG storage (path: \"$TEMP_DIR\")"
log_trace "<-- Lock $TEMP_STRG storage"
reset_trap
}
# ---
function unlock_storage {
set_trap "Unable to unlock ${1:-undefined} storage"
log_trace "--> Unlock ${1:-undefined} storage"
local FUNC_INIT="$($CMD_DATE "+%s%1N")"
# ---
local TEMP_STRG="${1:-undefined}"
if [[ "$TEMP_STRG" != "master" ]] && [[ "$TEMP_STRG" != "remote" ]]; then
exit_critical "Invalid unlock_storage call (strg: \"$TEMP_STRG\") - please send a bug report"
return
fi
# --- Not locked
if [[ "$TEMP_STRG" == "master" ]] && [[ -z "${MASTER_LOCK:-}" ]]; then
log_info2 "Unable to unlock $TEMP_STRG storage (MASTER_LOCK is unset) - ignoring"
log_trace "<-- Unlock $TEMP_STRG storage"
reset_trap
return 0
elif [[ "$TEMP_STRG" == "remote" ]] && [[ -z "${REMOTE_LOCK:-}" ]]; then
log_info2 "Unable to unlock $TEMP_STRG storage (REMOTE_LOCK is unset) - ignoring"
log_trace "<-- Unlock $TEMP_STRG storage"
reset_trap
return 0
fi
# --- Set lock directory
if [[ "$TEMP_STRG" == "master" ]]; then
local TEMP_DIR="$MASTER_DIR_PATH/$BACKUP_CTID/lock"
elif [[ "$TEMP_STRG" == "remote" ]]; then
local TEMP_DIR="$REMOTE_DIR_PATH/$BACKUP_CTID/lock"
fi
# --- Delete lock directory
if [[ "$OPTION_DRYRUN" -eq 1 ]]; then
local TASK_INIT="$($CMD_DATE "+%s%1N")"
if [[ "$TEMP_STRG" == "master" ]]; then
dry_run "$MASTER_SSH_OPTIONS \"$MASTER_SSH_HOST\" \"$CMD_RMDIR $ARGS_CPMKMVRM \\\"$TEMP_DIR/\\\"\""
elif [[ "$TEMP_STRG" == "remote" ]]; then
dry_run "$MASTER_SSH_OPTIONS \"$MASTER_SSH_HOST\" \"$REMOTE_SSH_OPTIONS \\\"$REMOTE_SSH_HOST\\\" \\\"$CMD_RMDIR $ARGS_CPMKMVRM \\\\\"$TEMP_DIR/\\\\\"\\\"\""
fi
local TASK_STOP="$($CMD_DATE "+%s%1N")"
else
local TASK_INIT="$($CMD_DATE "+%s%1N")"
set +o errexit; trap - ERR
if [[ "$TEMP_STRG" == "master" ]]; then
$MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "$CMD_RMDIR $ARGS_CPMKMVRM \"$TEMP_DIR/\"" 2>&1 | $CMD_TEE -a "$LOG_FILE"
elif [[ "$TEMP_STRG" == "remote" ]]; then
$MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "$REMOTE_SSH_OPTIONS \"$REMOTE_SSH_HOST\" \"$CMD_RMDIR $ARGS_CPMKMVRM \\\"$TEMP_DIR/\\\"\"" 2>&1 | $CMD_TEE -a "$LOG_FILE"
fi
local ERROR_CODE="$?"
set_trap "Unable to unlock $TEMP_STRG storage"
local TASK_STOP="$($CMD_DATE "+%s%1N")"
# --- Check error code
if [[ "$ERROR_CODE" -ne 0 ]]; then
if [[ "$ERROR_CODE" -eq 255 ]]; then
if [[ "$TEMP_STRG" == "master" ]]; then
exit_critical "SSH error: connection failure (host: \"$MASTER_SSH_HOST\" - error: \"$ERROR_CODE\") - aborting"
return
elif [[ "$TEMP_STRG" == "remote" ]]; then
exit_critical "SSH error: connection failure (host: \"$MASTER_SSH_HOST\" -> \"$REMOTE_SSH_HOST\" - error: \"$ERROR_CODE\") - aborting"
return
fi
fi
# Note: ALL (unlock_storage: fallback)
log_warning "Unable to unlock $TEMP_STRG storage (path: \"$TEMP_DIR\" - error: \"$ERROR_CODE\") - please send a bug report if this happens again"
fi
fi
if [[ "$TEMP_STRG" == "master" ]]; then
# Note: mutable
unset MASTER_LOCK
elif [[ "$TEMP_STRG" == "remote" ]]; then
# Note: mutable
unset REMOTE_LOCK
fi
# ---
local FUNC_STOP="$($CMD_DATE "+%s%1N")"
compute_time "$TASK_INIT" "$TASK_STOP"
log_time "$DISPLAY_STAT_TIME_1ST - Unlock $TEMP_STRG storage (path: \"$TEMP_DIR\")"
log_trace "<-- Unlock $TEMP_STRG storage"
reset_trap
}
# ---
function update_task_file {
set_trap "Unable to update ${1:-undefined} task file"
log_trace "--> Update ${1:-undefined} task file"
local FUNC_INIT="$($CMD_DATE "+%s%1N")"
# ---
local TEMP_STRG="${1:-undefined}"
if [[ "$TEMP_STRG" != "master" ]] && [[ "$TEMP_STRG" != "remote" ]]; then
exit_critical "Invalid update_task_file call (strg: \"$TEMP_STRG\") - please send a bug report"
return
fi
# --- Not locked
if [[ "$TEMP_STRG" == "master" ]] && [[ -z "${MASTER_LOCK:-}" ]]; then
exit_critical "Unable to update $TEMP_STRG task file (MASTER_LOCK is unset) - please send a bug report"
return
elif [[ "$TEMP_STRG" == "remote" ]] && [[ -z "${REMOTE_LOCK:-}" ]]; then
exit_critical "Unable to update $TEMP_STRG task file (REMOTE_LOCK is unset) - please send a bug report"
return
fi
# --- Check task data
local TASK_DATA="${2:-undefined}"
declare -a TEMP_ARRAY="()"
IFS="|" read -a TEMP_ARRAY <<< "$TASK_DATA"
local TEMP_INDEX="0"
local TEMP_COUNT="${#TEMP_ARRAY[@]}"
for TEMP_NAME in TASK_NAME TASK_EXIT TASK_MODE TASK_HOST TASK_HASH TASK_KEEP; do
if [[ "$TEMP_INDEX" -lt "$TEMP_COUNT" ]]; then
local $TEMP_NAME="${TEMP_ARRAY[$TEMP_INDEX]}"
else
local $TEMP_NAME=""
fi
local TEMP_INDEX="$(( TEMP_INDEX + 1 ))"
done
# --- Check infos
if [[ "$TASK_NAME" != "backup" ]] && [[ "$TASK_NAME" != "delete" ]] && [[ "$TASK_NAME" != "destroy" ]] && [[ "$TASK_NAME" != "download" ]] && [[ "$TASK_NAME" != "upload" ]]; then
exit_critical "Invalid task name (data: \"$TASK_DATA\") - please send a bug report"
return
fi
if [[ "$TASK_EXIT" != "running" ]] && [[ "$TASK_EXIT" != "success" ]] && [[ "$TASK_EXIT" != "failure" ]]; then
exit_critical "Invalid task exit (data: \"$TASK_DATA\") - please send a bug report"
return
fi
if [[ ! "$TASK_MODE" =~ ^(cold|hold|live)_(sync|snap|plop)$ ]]; then
exit_critical "Invalid task mode (data: \"$TASK_DATA\") - please send a bug report"
return
fi
if [[ -z "$TASK_HOST" ]]; then
exit_critical "Invalid task host (data: \"$TASK_DATA\") - please send a bug report"
return
fi
if [[ "$TASK_HASH" != "unknown" ]] && [[ "${#TASK_HASH}" -ne 32 ]] && [[ "$TASK_HASH" == *[!0-9a-f]* ]]; then
exit_critical "Invalid task hash (data: \"$TASK_DATA\") - please send a bug report"
return
fi
if [[ "$TASK_KEEP" != "0000-00-00_00-00-00" ]]; then
local KEEPBK_TS="${TASK_KEEP:0:4}${TASK_KEEP:5:2}${TASK_KEEP:8:2}${TASK_KEEP:11:2}${TASK_KEEP:14:2}${TASK_KEEP:17:2}"
if [[ "${#TASK_KEEP}" -ne 19 ]] || [[ "${#KEEPBK_TS}" -ne 14 ]] || [[ "$KEEPBK_TS" == *[!0-9]* ]] || [[ "$KEEPBK_TS" != "${TASK_KEEP//[\-_]/}" ]]; then
# [[ "$($CMD_DATE -d "${TASK_KEEP:0:4}-${TASK_KEEP:5:2}-${TASK_KEEP:8:2} ${TASK_KEEP:11:2}:${TASK_KEEP:14:2}:${TASK_KEEP:17:2}" "+%Y-%m-%d_%H-%M-%S")" != "$TASK_KEEP" ]]; then
exit_critical "Invalid task keep (data: \"$TASK_DATA\") - please send a bug report"
return
fi
fi
# --- Set temp file
if [[ "$TEMP_STRG" == "master" ]]; then
local TEMP_PATH="$MASTER_DIR_PATH/$BACKUP_CTID/task"
elif [[ "$TEMP_STRG" == "remote" ]]; then
local TEMP_PATH="$REMOTE_DIR_PATH/$BACKUP_CTID/task"
fi
local TEMP_FILE="$TEMP_PATH/task_${BACKUP_CTID}_${BACKUP_DATE}.txt"
# --- Update task file
if [[ "$OPTION_DRYRUN" -eq 1 ]]; then
local TASK_INIT="$($CMD_DATE "+%s%1N")"
if [[ "$TEMP_STRG" == "master" ]]; then
dry_run "$MASTER_SSH_OPTIONS \"$MASTER_SSH_HOST\" \" \
$CMD_ECHO \\\"$TASK_DATA\\\" > \\\"$TEMP_FILE\\\" && \
$CMD_CHOWN $ARGS_CHOWNMOD \\\"root:root\\\" \\\"$TEMP_FILE\\\" && \
$CMD_CHMOD $ARGS_CHOWNMOD \\\"0640\\\" \\\"$TEMP_FILE\\\" && \
$CMD_TOUCH \\\"$TEMP_PATH/\\\" \
\""
elif [[ "$TEMP_STRG" == "remote" ]]; then
dry_run "$MASTER_SSH_OPTIONS \"$MASTER_SSH_HOST\" \"$REMOTE_SSH_OPTIONS \\\"$REMOTE_SSH_HOST\\\" \\\" \
$CMD_ECHO \\\\\"$TASK_DATA\\\\\" > \\\\\"$TEMP_FILE\\\\\" && \
$CMD_CHOWN $ARGS_CHOWNMOD \\\\\"root:root\\\\\" \\\\\"$TEMP_FILE\\\\\" && \
$CMD_CHMOD $ARGS_CHOWNMOD \\\\\"0640\\\\\" \\\\\"$TEMP_FILE\\\\\" && \
$CMD_TOUCH \\\\\"$TEMP_PATH/\\\\\" \
\\\"\""
fi
local TASK_STOP="$($CMD_DATE "+%s%1N")"
else
local TASK_INIT="$($CMD_DATE "+%s%1N")"
set +o errexit; trap - ERR
if [[ "$TEMP_STRG" == "master" ]]; then
$MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" " \
$CMD_ECHO \"$TASK_DATA\" > \"$TEMP_FILE\" && \
$CMD_CHOWN $ARGS_CHOWNMOD \"root:root\" \"$TEMP_FILE\" && \
$CMD_CHMOD $ARGS_CHOWNMOD \"0640\" \"$TEMP_FILE\" && \
$CMD_TOUCH \"$TEMP_PATH/\" \
" 2>&1 | $CMD_TEE -a "$LOG_FILE"
elif [[ "$TEMP_STRG" == "remote" ]]; then
$MASTER_SSH_OPTIONS "$MASTER_SSH_HOST" "$REMOTE_SSH_OPTIONS \"$REMOTE_SSH_HOST\" \" \
$CMD_ECHO \\\"$TASK_DATA\\\" > \\\"$TEMP_FILE\\\" && \
$CMD_CHOWN $ARGS_CHOWNMOD \\\"root:root\\\" \\\"$TEMP_FILE\\\" && \
$CMD_CHMOD $ARGS_CHOWNMOD \\\"0640\\\" \\\"$TEMP_FILE\\\" && \
$CMD_TOUCH \\\"$TEMP_PATH/\\\" \
\"" 2>&1 | $CMD_TEE -a "$LOG_FILE"
fi
local ERROR_CODE="$?"
set_trap "Unable to update $TEMP_STRG task file"
local TASK_STOP="$($CMD_DATE "+%s%1N")"
# --- Check error code
if [[ "$ERROR_CODE" -ne 0 ]]; then
if [[ "$ERROR_CODE" -eq 255 ]]; then
if [[ "$TEMP_STRG" == "master" ]]; then
exit_critical "SSH error: connection failure (host: \"$MASTER_SSH_HOST\" - error: \"$ERROR_CODE\") - aborting"
return
elif [[ "$TEMP_STRG" == "remote" ]]; then
exit_critical "SSH error: connection failure (host: \"$MASTER_SSH_HOST\" -> \"$REMOTE_SSH_HOST\" - error: \"$ERROR_CODE\") - aborting"
return
fi
fi
exit_error "Unable to update $TEMP_STRG task file (path: \"$TEMP_FILE\" - data: \"$TASK_DATA\" - error: \"$ERROR_CODE\") - aborting"
return
fi
fi
if [[ "$TEMP_STRG" == "master" ]]; then
# Note: mutable
MASTER_TASK="$TASK_DATA"
elif [[ "$TEMP_STRG" == "remote" ]]; then
# Note: mutable
REMOTE_TASK="$TASK_DATA"
fi
# ---
local FUNC_STOP="$($CMD_DATE "+%s%1N")"
compute_time "$TASK_INIT" "$TASK_STOP"
log_time "$DISPLAY_STAT_TIME_1ST - Update $TEMP_STRG task file (data: \"$TASK_DATA\")"
log_trace "<-- Update $TEMP_STRG task file"
reset_trap
}
# ---
function create_container {
set_trap "Unable to create container"
log_trace "--> Create container"
local FUNC_INIT="$($CMD_DATE "+%s%1N")"
# ---
if [[ -z "$TARGET_TYPE" ]] || ( [[ "$TARGET_TYPE" != "simfs" ]] && [[ "$TARGET_TYPE" != "ploop" ]] ); then
exit_critical "Invalid container type (type: \"$TARGET_TYPE\" - size: \"$TARGET_SIZE\" - inod: \"$TARGET_INOD\" - priv: \"${TARGET_PRIV%/*}\" - root: \"${TARGET_ROOT%/*}\" - tmpl: \"$TARGET_TMPL\") - please send a bug report"
return
fi
if [[ -z "$TARGET_SIZE" ]] || [[ ! "$TARGET_SIZE" =~ ^[0-9]+:[0-9]+$ ]]; then
exit_critical "Invalid container size (type: \"$TARGET_TYPE\" - size: \"$TARGET_SIZE\" - inod: \"$TARGET_INOD\" - priv: \"${TARGET_PRIV%/*}\" - root: \"${TARGET_ROOT%/*}\" - tmpl: \"$TARGET_TMPL\") - please send a bug report"
return
fi
if [[ -z "$TARGET_INOD" ]] || [[ ! "$TARGET_INOD" =~ ^[0-9]+:[0-9]+$ ]]; then
exit_critical "Invalid container inod (type: \"$TARGET_TYPE\" - size: \"$TARGET_SIZE\" - inod: \"$TARGET_INOD\" - priv: \"${TARGET_PRIV%/*}\" - root: \"${TARGET_ROOT%/*}\" - tmpl: \"$TARGET_TMPL\") - please send a bug report"
return
fi
if [[ -z "$TARGET_PRIV" ]] || [[ ! -d "${TARGET_PRIV%/*}/" ]]; then
exit_critical "Invalid container priv (type: \"$TARGET_TYPE\" - size: \"$TARGET_SIZE\" - inod: \"$TARGET_INOD\" - priv: \"${TARGET_PRIV%/*}\" - root: \"${TARGET_ROOT%/*}\" - tmpl: \"$TARGET_TMPL\") - please send a bug report"
return
fi
if [[ -z "$TARGET_ROOT" ]] || [[ ! -d "${TARGET_ROOT%/*}/" ]]; then
exit_critical "Invalid container root (type: \"$TARGET_TYPE\" - size: \"$TARGET_SIZE\" - inod: \"$TARGET_INOD\" - priv: \"${TARGET_PRIV%/*}\" - root: \"${TARGET_ROOT%/*}\" - tmpl: \"$TARGET_TMPL\") - please send a bug report"
return
fi
if [[ -z "$TARGET_TMPL" ]]; then
exit_critical "Invalid container tmpl (type: \"$TARGET_TYPE\" - size: \"$TARGET_SIZE\" - inod: \"$TARGET_INOD\" - priv: \"${TARGET_PRIV%/*}\" - root: \"${TARGET_ROOT%/*}\" - tmpl: \"$TARGET_TMPL\") - please send a bug report"
return
fi
# --- task_restore
if [[ -n "${TARGET_CTID:-}" ]] && [[ "$TARGET_CTID" != "$BACKUP_CTID" ]]; then
log_debug "Switching backup/target CTID (backup: $BACKUP_CTID - target: $TARGET_CTID)"
local TASK_CTID="$TARGET_CTID"
else
local TASK_CTID="$BACKUP_CTID"
fi
# --- Check status
check_container_status
if [[ "$CT_STATUS_EXIST" != "deleted" ]]; then
exit_critical "Container already exists (ctid: $TASK_CTID - status: \"$CT_STATUS\") - please send a bug report"
return
fi
# --- Create container
log_info "Creating container (ctid: $TASK_CTID - diskspace: \"$TARGET_SIZE\" - diskinodes: \"$TARGET_INOD\" - private: \"$TARGET_PRIV\" - root: \"$TARGET_ROOT\" - ostemplate \"$TARGET_TMPL\" - layout: \"$TARGET_TYPE\")"
if [[ "$OPTION_DRYRUN" -eq 1 ]]; then
local TASK_INIT="$($CMD_DATE "+%s%1N")"
if [[ "$CONFIG_TYPE" == "Proxmox" ]]; then
# FIXME (low): hardcoded (PVE2/PVE3 - /usr/lib/vzctl/scripts/vps-download)
dry_run "/usr/lib/vzctl/scripts/vps-download \"$TARGET_TMPL\""
if [[ -n "${TARGET_STORAGE_NAME:-}" ]]; then
dry_run "$CMD_PVECTL create \"$TARGET_CTID\" \"${TARGET_TDIR}/cache/$TARGET_TMPL.tar.gz\" -storage \"$TARGET_STORAGE_NAME\""
else
dry_run "$CMD_PVECTL create \"$TARGET_CTID\" \"${TARGET_TDIR}/cache/$TARGET_TMPL.tar.gz\""
fi
elif [[ "$CONFIG_TYPE" == "Virtuozzo" ]]; then
dry_run "$CMD_VZCTL $ARGS_VZCTL create \"$TASK_CTID\" --diskspace \"$TARGET_SIZE\" --diskinodes \"$TARGET_INOD\" --private \"$TARGET_PRIV\" --root \"$TARGET_ROOT\" --ostemplate \"$TARGET_TMPL\""
elif [[ "$CONFIG_TYPE" == "OpenVZ" ]]; then
dry_run "$CMD_VZCTL $ARGS_VZCTL create \"$TASK_CTID\" --diskspace \"$TARGET_SIZE\" --diskinodes \"$TARGET_INOD\" --private \"$TARGET_PRIV\" --root \"$TARGET_ROOT\" --ostemplate \"$TARGET_TMPL\" --layout \"$TARGET_TYPE\""
else
exit_critical "Invalid configuration type (OpenVZ/Virtuozzo/Proxmox) - please send a bug report"
return
fi
local TASK_STOP="$($CMD_DATE "+%s%1N")"
else
local TASK_INIT="$($CMD_DATE "+%s%1N")"
set +o errexit; trap - ERR
if [[ "$CONFIG_TYPE" == "Proxmox" ]]; then
if [[ "$ARGS_VZCTL" == "--quiet" ]]; then
# FIXME (low): hardcoded (PVE2/PVE3 - /usr/lib/vzctl/scripts/vps-download)
/usr/lib/vzctl/scripts/vps-download "$TARGET_TMPL" 2>> "$LOG_FILE" > "/dev/null"
if [[ -n "${TARGET_STORAGE_NAME:-}" ]]; then
$CMD_PVECTL create "$TARGET_CTID" "${TARGET_TDIR}/cache/$TARGET_TMPL.tar.gz" -storage "$TARGET_STORAGE_NAME" 2>> "$LOG_FILE" > "/dev/null"
else
$CMD_PVECTL create "$TARGET_CTID" "${TARGET_TDIR}/cache/$TARGET_TMPL.tar.gz" 2>> "$LOG_FILE" > "/dev/null"
fi
else
# FIXME (low): hardcoded (PVE2/PVE3 - /usr/lib/vzctl/scripts/vps-download)
/usr/lib/vzctl/scripts/vps-download "$TARGET_TMPL" 2>&1 | $CMD_TEE -a "$LOG_FILE"
if [[ -n "${TARGET_STORAGE_NAME:-}" ]]; then
$CMD_PVECTL create "$TARGET_CTID" "${TARGET_TDIR}/cache/$TARGET_TMPL.tar.gz" -storage "$TARGET_STORAGE_NAME" 2>&1 | $CMD_TEE -a "$LOG_FILE"
else
$CMD_PVECTL create "$TARGET_CTID" "${TARGET_TDIR}/cache/$TARGET_TMPL.tar.gz" 2>&1 | $CMD_TEE -a "$LOG_FILE"
fi
fi
elif [[ "$CONFIG_TYPE" == "Virtuozzo" ]]; then
if [[ "$ARGS_VZCTL" == "--quiet" ]]; then
$CMD_VZCTL create "$TASK_CTID" --diskspace "$TARGET_SIZE" --diskinodes "$TARGET_INOD" --private "$TARGET_PRIV" --root "$TARGET_ROOT" --ostemplate "$TARGET_TMPL" 2>> "$LOG_FILE" > "/dev/null"
else
$CMD_VZCTL $ARGS_VZCTL create "$TASK_CTID" --diskspace "$TARGET_SIZE" --diskinodes "$TARGET_INOD" --private "$TARGET_PRIV" --root "$TARGET_ROOT" --ostemplate "$TARGET_TMPL" 2>&1 | $CMD_TEE -a "$LOG_FILE"
fi
elif [[ "$CONFIG_TYPE" == "OpenVZ" ]]; then
if [[ "$ARGS_VZCTL" == "--quiet" ]]; then
$CMD_VZCTL create "$TASK_CTID" --diskspace "$TARGET_SIZE" --diskinodes "$TARGET_INOD" --private "$TARGET_PRIV" --root "$TARGET_ROOT" --ostemplate "$TARGET_TMPL" --layout "$TARGET_TYPE" 2>> "$LOG_FILE" > "/dev/null"
else
$CMD_VZCTL $ARGS_VZCTL create "$TASK_CTID" --diskspace "$TARGET_SIZE" --diskinodes "$TARGET_INOD" --private "$TARGET_PRIV" --root "$TARGET_ROOT" --ostemplate "$TARGET_TMPL" --layout "$TARGET_TYPE" 2>&1 | $CMD_TEE -a "$LOG_FILE"
fi
else
exit_critical "Invalid configuration type (OpenVZ/Virtuozzo/Proxmox) - please send a bug report"
return
fi
local ERROR_CODE="$?"
set_trap "Unable to create container"
local TASK_STOP="$($CMD_DATE "+%s%1N")"
# --- Check error code
if [[ "$ERROR_CODE" -ne 0 ]]; then
exit_error "Unable to create container (ctid: $TASK_CTID - diskspace: \"$TARGET_SIZE\" - diskinodes: \"$TARGET_INOD\" - private: \"$TARGET_PRIV\" - root: \"$TARGET_ROOT\" - ostemplate \"$TARGET_TMPL\" - layout: \"$TARGET_TYPE\" - error: \"$ERROR_CODE\") - aborting"
return
fi
fi
# ---
local FUNC_STOP="$($CMD_DATE "+%s%1N")"
compute_time "$TASK_INIT" "$TASK_STOP"
log_time "$DISPLAY_STAT_TIME_1ST - Create container (ctid: $TASK_CTID)"
log_trace "<-- Create container"
reset_trap
}
# ---
function start_container {
set_trap "Unable to start container"
log_trace "--> Start container"
local FUNC_INIT="$($CMD_DATE "+%s%1N")"
# Note: mutable
unset DOWN_STOP
# --- task_restore
if [[ -n "${TARGET_CTID:-}" ]] && [[ "$TARGET_CTID" != "$BACKUP_CTID" ]]; then
log_debug "Switching backup/target CTID (backup: $BACKUP_CTID - target: $TARGET_CTID)"
local TASK_CTID="$TARGET_CTID"
else
local TASK_CTID="$BACKUP_CTID"
fi
# --- Check status
check_container_status
if [[ "$CT_STATUS_EXIST" != "exist" ]]; then
exit_critical "Container not found (ctid: $TASK_CTID - status: \"$CT_STATUS\") - please send a bug report"
return
fi
# --- Start container
if [[ "$OPTION_DRYRUN" -eq 1 ]]; then
execute_hook "before_start_container"
local TASK_INIT="$($CMD_DATE "+%s%1N")"
dry_run "$CMD_VZCTL $ARGS_VZCTL start \"$TASK_CTID\""
local TASK_STOP="$($CMD_DATE "+%s%1N")"
# Note: mutable
DOWN_STOP="$TASK_STOP"
if [[ "$(( OPTION_OPTIMIZE & 2 ))" -eq 2 ]]; then
log_debug "Optimized mode ( 2) - container is starting - waiting 10 seconds (dry-run mode)"
task_pause 10 force
fi
log_debug "Container has been started (dry-run mode)"
execute_hook "after_start_container"
else
if [[ "$CT_STATUS_MOUNT" == "mounted" ]] && [[ "$CT_STATUS_RUNNING" == "running" ]] && [[ "$CT_STATUS_SUSPENDED" == "" ]]; then
# Note: ALL (start_container: fallback)
log_warning "Container already started (ctid: $TASK_CTID - status: \"$CT_STATUS\") - please send a bug report if this happens again"
execute_hook "after_start_container"
log_trace "<-- Start container"
reset_trap
return 0
fi
# Note: ALL (start_container: vzctl stop <ctid> => --skip-umount)
if [[ "$CT_STATUS_MOUNT" != "mounted" ]] || [[ "$CT_STATUS_RUNNING" != "down" ]] || [[ "$CT_STATUS_SUSPENDED" != "" ]]; then
exit_critical "Unexpected container state (ctid: $TASK_CTID - status: \"$CT_STATUS\" - expected: \"exist mounted down\") - please send a bug report"
return
fi
# ---
execute_hook "before_start_container"
local TASK_INIT="$($CMD_DATE "+%s%1N")"
set +o errexit; trap - ERR
# Note: ALL (start_container: vzctl start <ctid> --skip-remount" option => MAJOR BUG with bind mounts)
if [[ "$ARGS_VZCTL" == "--quiet" ]]; then
$CMD_VZCTL start "$TASK_CTID" 2>> "$LOG_FILE" > "/dev/null"
else
$CMD_VZCTL $ARGS_VZCTL start "$TASK_CTID" 2>&1 | $CMD_TEE -a "$LOG_FILE"
fi
local ERROR_CODE="$?"
set_trap "Unable to start container"
local TASK_STOP="$($CMD_DATE "+%s%1N")"
# Note: mutable
DOWN_STOP="$TASK_STOP"
# --- Check error code
if [[ "$ERROR_CODE" -ne 0 ]]; then
check_container_status
exit_error "Unable to start container (ctid: $TASK_CTID - status: \"$CT_STATUS\" - error: \"$ERROR_CODE\") - aborting"
return
fi
# ---
if [[ "$(( OPTION_OPTIMIZE & 2 ))" -eq 2 ]]; then
log_info2 "Optimized mode ( 2) - container is starting - waiting 10 seconds"
task_pause 10 force
fi
log_info2 "Container has been started - use \"$CMD_VZCTL stop $TASK_CTID\" to stop it manually"
execute_hook "after_start_container"
fi
# ---
local FUNC_STOP="$($CMD_DATE "+%s%1N")"
compute_time "$TASK_INIT" "$TASK_STOP"
log_time "$DISPLAY_STAT_TIME_1ST - Start container (ctid: $TASK_CTID)"
log_trace "<-- Start container"
reset_trap
}
# ---
function stop_container {
set_trap "Unable to stop container"
log_trace "--> Stop container"
local FUNC_INIT="$($CMD_DATE "+%s%1N")"
# Note: mutable
unset DOWN_INIT
# --- task_restore
if [[ -n "${TARGET_CTID:-}" ]] && [[ "$TARGET_CTID" != "$BACKUP_CTID" ]]; then
log_debug "Switching backup/target CTID (backup: $BACKUP_CTID - target: $TARGET_CTID)"
local TASK_CTID="$TARGET_CTID"
else
local TASK_CTID="$BACKUP_CTID"
fi
# --- Check status
check_container_status
if [[ "$CT_STATUS_EXIST" != "exist" ]]; then
exit_critical "Container not found (ctid: $TASK_CTID - status: \"$CT_STATUS\") - please send a bug report"
return
fi
# --- Stop container
if [[ "$OPTION_DRYRUN" -eq 1 ]]; then
execute_hook "before_stop_container"
local TASK_INIT="$($CMD_DATE "+%s%1N")"
# Note: mutable
DOWN_INIT="$TASK_INIT"
dry_run "$CMD_VZCTL $ARGS_VZCTL stop \"$TASK_CTID\" --skip-umount"
local TASK_STOP="$($CMD_DATE "+%s%1N")"
log_debug "Container has been stopped (dry-run mode)"
execute_hook "after_stop_container"
else
# Note: ALL (stop_container: vzctl stop <ctid> => --skip-umount)
if [[ "$CT_STATUS_MOUNT" == "mounted" ]] && [[ "$CT_STATUS_RUNNING" == "down" ]] && [[ "$CT_STATUS_SUSPENDED" == "" ]]; then
# Note: ALL (stop_container: fallback)
log_warning "Container already stopped (ctid: $TASK_CTID - status: \"$CT_STATUS\") - please send a bug report if this happens again"
execute_hook "after_stop_container"
log_trace "<-- Stop container"
reset_trap
return 0
fi
if [[ "$CT_STATUS_MOUNT" != "mounted" ]] || [[ "$CT_STATUS_RUNNING" != "running" ]] || [[ "$CT_STATUS_SUSPENDED" != "" ]]; then
exit_critical "Unexpected container state (ctid: $TASK_CTID - status: \"$CT_STATUS\" - expected: \"exist mounted running\") - please send a bug report"
return
fi
# ---
execute_hook "before_stop_container"
local TASK_INIT="$($CMD_DATE "+%s%1N")"
# Note: mutable
DOWN_INIT="$TASK_INIT"
set +o errexit; trap - ERR
if [[ "$ARGS_VZCTL" == "--quiet" ]]; then
$CMD_VZCTL stop "$TASK_CTID" --skip-umount 2>> "$LOG_FILE" > "/dev/null"
else
$CMD_VZCTL $ARGS_VZCTL stop "$TASK_CTID" --skip-umount 2>&1 | $CMD_TEE -a "$LOG_FILE"
fi
local ERROR_CODE="$?"
set_trap "Unable to stop container"
local TASK_STOP="$($CMD_DATE "+%s%1N")"
# --- Check error code
if [[ "$ERROR_CODE" -ne 0 ]]; then
check_container_status
exit_error "Unable to stop container (ctid: $TASK_CTID - status: \"$CT_STATUS\" - error: \"$ERROR_CODE\") - aborting"
return
fi
# ---
log_info2 "Container has been stopped - use \"$CMD_VZCTL start $TASK_CTID\" to start it manually"
execute_hook "after_stop_container"
fi
# ---
local FUNC_STOP="$($CMD_DATE "+%s%1N")"
compute_time "$TASK_INIT" "$TASK_STOP"
log_time "$DISPLAY_STAT_TIME_1ST - Stop container (ctid: $TASK_CTID)"
log_trace "<-- Stop container"
reset_trap
}
# ---
function suspend_container {
set_trap "Unable to suspend container"
log_trace "--> Suspend container"
local FUNC_INIT="$($CMD_DATE "+%s%1N")"
# Note: mutable
unset DOWN_INIT
# --- task_restore
if [[ -n "${TARGET_CTID:-}" ]] && [[ "$TARGET_CTID" != "$BACKUP_CTID" ]]; then
log_debug "Switching backup/target CTID (backup: $BACKUP_CTID - target: $TARGET_CTID)"
local TASK_CTID="$TARGET_CTID"
else
local TASK_CTID="$BACKUP_CTID"
fi
# --- Check status
check_container_status
if [[ "$CT_STATUS_EXIST" != "exist" ]]; then
exit_critical "Container not found (ctid: $TASK_CTID - status: \"$CT_STATUS\") - please send a bug report"
return
fi
# --- Suspend container
if [[ "$OPTION_DRYRUN" -eq 1 ]]; then
if [[ "$CONFIG_TYPE" != "Virtuozzo" ]]; then
execute_hook "before_suspend_container"
fi
local TASK_INIT="$($CMD_DATE "+%s%1N")"
# Note: mutable
DOWN_INIT="$TASK_INIT"
if [[ "$CONFIG_TYPE" != "Virtuozzo" ]]; then
dry_run "$CMD_VZCTL $ARGS_VZCTL chkpnt \"$TASK_CTID\" --suspend"
else
log_info2 "$CONFIG_TYPE ${CONFIG_XVER:-Legacy} detected - unable to suspend container (operation not supported)"
fi
local TASK_STOP="$($CMD_DATE "+%s%1N")"
if [[ "$CONFIG_TYPE" != "Virtuozzo" ]]; then
log_debug "Container has been suspended (dry-run mode)"
execute_hook "after_suspend_container"
fi
else
if [[ "$CONFIG_TYPE" != "Virtuozzo" ]]; then
if [[ "$CT_STATUS_MOUNT" == "mounted" ]] && [[ "$CT_STATUS_RUNNING" == "running" ]] && [[ "$CT_STATUS_SUSPENDED" == "suspended" ]]; then
# Note: ALL (suspend_container: fallback)
log_warning "Container already suspended (ctid: $TASK_CTID - status: \"$CT_STATUS\") - please send a bug report if this happens again"
execute_hook "after_suspend_container"
log_trace "<-- Suspend container"
reset_trap
return 0
fi
if [[ "$CT_STATUS_MOUNT" != "mounted" ]] || [[ "$CT_STATUS_RUNNING" != "running" ]] || [[ "$CT_STATUS_SUSPENDED" != "" ]]; then
exit_critical "Unexpected container state (ctid: $TASK_CTID - status: \"$CT_STATUS\" - expected: \"exist mounted running\") - please send a bug report"
return
fi
# ---
if [[ "$KICK_VZCTL_PROCESS" == "yes" ]]; then
log_debug "Looking for \"$CMD_VZCTL\" process inside container (ctid: $TASK_CTID - status: \"$CT_STATUS\")"
local VZCTL_LIST="$($CMD_PGREP -G "root" -U "root" "$CMD_VZCTL" 2>> "$LOG_FILE")"
# local VZCTL_LIST="${VZCTL_LIST//$'\r'/ }"
local VZCTL_LIST="${VZCTL_LIST//$'\n'/ }"
# local VZCTL_LIST="${VZCTL_LIST// / }"
local VZCTL_PID=""
for VZCTL_PID in $VZCTL_LIST; do
local VZCTL_CTID="$($CMD_VZPID "$VZCTL_PID" 2>> "$LOG_FILE" | $CMD_TAIL -n "1" | $CMD_CUT -f "2" -s)"
if [[ -z "$VZCTL_CTID" ]] || [[ "$VZCTL_CTID" == "CTID" ]] || [[ "$VZCTL_CTID" == "VEID" ]]; then
log_info2 "\"$CMD_VZCTL\" process has vanished (ctid: $VZCTL_CTID - pid: \"$VZCTL_PID\" - list: \"$VZCTL_LIST\") - ignoring"
continue
elif [[ "$VZCTL_CTID" == *[!0-9]* ]]; then
# Note: ALL (suspend_container: fallback)
log_warning "Unexpected \"$CMD_VZCTL\" process (ctid: $VZCTL_CTID - pid: \"$VZCTL_PID\" - list: \"$VZCTL_LIST\") - please send a bug report if this happens again"
continue
elif [[ "$VZCTL_CTID" != "$TASK_CTID" ]]; then
log_debug "Foreign \"$CMD_VZCTL\" process (ctid: $VZCTL_CTID - pid: \"$VZCTL_PID\") - skipping"
continue
fi
# ---
log_info2 "Sending \"HUP\" signal to \"$CMD_VZCTL\" process found inside container (ctid: $VZCTL_CTID - pid: \"$VZCTL_PID\")"
task_pause
set +o errexit; trap - ERR
$CMD_KILL -HUP "$VZCTL_PID" 2>&1 | $CMD_TEE -a "$LOG_FILE"
local ERROR_CODE="$?"
set_trap "Unable to suspend container"
if [[ "$ERROR_CODE" -ne 0 ]]; then
# Note: ALL (suspend_container: fallback)
log_warning "Unable to send \"HUP\" signal to \"$CMD_VZCTL\" process (ctid: $VZCTL_CTID - pid: \"$VZCTL_PID\") - please send a bug report if this happens again"
fi
done
fi
# ---
execute_hook "before_suspend_container"
# --- Increase cpt/rst debug level
if [[ "$OPTION_VERBOSE" -gt 0 ]]; then
if [[ -f "/proc/sys/debug/cpt" ]]; then
local TEMP_PROC_CPT="$($CMD_CAT "/proc/sys/debug/cpt")"
$CMD_ECHO "$(( OPTION_VERBOSE + 1 ))" > "/proc/sys/debug/cpt"
fi
if [[ -f "/proc/sys/debug/rst" ]]; then
local TEMP_PROC_RST="$($CMD_CAT "/proc/sys/debug/rst")"
$CMD_ECHO "$(( OPTION_VERBOSE + 1 ))" > "/proc/sys/debug/rst"
fi
fi
fi
# ---
local TASK_INIT="$($CMD_DATE "+%s%1N")"
# Note: mutable
DOWN_INIT="$TASK_INIT"
set +o errexit; trap - ERR
if [[ "$CONFIG_TYPE" != "Virtuozzo" ]]; then
if [[ "$ARGS_VZCTL" == "--quiet" ]]; then
$CMD_VZCTL chkpnt "$TASK_CTID" --suspend 2>> "$LOG_FILE" > "/dev/null"
else
$CMD_VZCTL $ARGS_VZCTL chkpnt "$TASK_CTID" --suspend 2>&1 | $CMD_TEE -a "$LOG_FILE"
fi
else
log_info2 "$CONFIG_TYPE ${CONFIG_XVER:-Legacy} detected - unable to suspend container (operation not supported)"
fi
local ERROR_CODE="$?"
set_trap "Unable to suspend container"
local TASK_STOP="$($CMD_DATE "+%s%1N")"
# --- Restore cpt/rst debug level
if [[ "$CONFIG_TYPE" != "Virtuozzo" ]]; then
if [[ "$OPTION_VERBOSE" -gt 0 ]]; then
if [[ -f "/proc/sys/debug/cpt" ]]; then
$CMD_ECHO "$TEMP_PROC_CPT" > "/proc/sys/debug/cpt"
fi
if [[ -f "/proc/sys/debug/rst" ]]; then
$CMD_ECHO "$TEMP_PROC_RST" > "/proc/sys/debug/rst"
fi
fi
fi
# --- Check error code
if [[ "$ERROR_CODE" -ne 0 ]]; then
check_container_status
exit_error "Unable to suspend container (ctid: $TASK_CTID - status: \"$CT_STATUS\" - error: \"$ERROR_CODE\") - aborting"
return
fi
# ---
if [[ "$CONFIG_TYPE" != "Virtuozzo" ]]; then
log_info2 "Container has been suspended - use \"$CMD_VZCTL chkpnt $TASK_CTID --resume\" to resume it manually"
execute_hook "after_suspend_container"
fi
fi
# ---
local FUNC_STOP="$($CMD_DATE "+%s%1N")"
compute_time "$TASK_INIT" "$TASK_STOP"
log_time "$DISPLAY_STAT_TIME_1ST - Suspend container (ctid: $TASK_CTID)"
log_trace "<-- Suspend container"
reset_trap
}
# ---
function resume_container {
set_trap "Unable to resume container"
log_trace "--> Resume container"
local FUNC_INIT="$($CMD_DATE "+%s%1N")"
# Note: mutable
unset DOWN_STOP
# --- task_restore
if [[ -n "${TARGET_CTID:-}" ]] && [[ "$TARGET_CTID" != "$BACKUP_CTID" ]]; then
log_debug "Switching backup/target CTID (backup: $BACKUP_CTID - target: $TARGET_CTID)"
local TASK_CTID="$TARGET_CTID"
else
local TASK_CTID="$BACKUP_CTID"
fi
# --- Check status
check_container_status
if [[ "$CT_STATUS_EXIST" != "exist" ]]; then
exit_critical "Container not found (ctid: $TASK_CTID - status: \"$CT_STATUS\") - please send a bug report"
return
fi
# --- Resume container
if [[ "$OPTION_DRYRUN" -eq 1 ]]; then
if [[ "$CONFIG_TYPE" != "Virtuozzo" ]]; then
execute_hook "before_resume_container"
fi
local TASK_INIT="$($CMD_DATE "+%s%1N")"
if [[ "$CONFIG_TYPE" != "Virtuozzo" ]]; then
dry_run "$CMD_VZCTL $ARGS_VZCTL chkpnt \"$TASK_CTID\" --resume"
else
log_info2 "$CONFIG_TYPE ${CONFIG_XVER:-Legacy} detected - unable to resume container (operation not supported)"
fi
local TASK_STOP="$($CMD_DATE "+%s%1N")"
# Note: mutable
DOWN_STOP="$TASK_STOP"
if [[ "$CONFIG_TYPE" != "Virtuozzo" ]]; then
if [[ "$(( OPTION_OPTIMIZE & 2 ))" -eq 2 ]]; then
log_debug "Optimized mode ( 2) - container is resuming - waiting 5 seconds (dry-run mode)"
task_pause 5 force
fi
log_debug "Container has been resumed (dry-run mode)"
execute_hook "after_resume_container"
fi
else
if [[ "$CONFIG_TYPE" != "Virtuozzo" ]]; then
if [[ "$CT_STATUS_MOUNT" == "mounted" ]] && [[ "$CT_STATUS_RUNNING" == "running" ]] && [[ "$CT_STATUS_SUSPENDED" == "" ]]; then
# Note: ALL (resume_container: fallback)
log_warning "Container already resumed (ctid: $TASK_CTID - status: \"$CT_STATUS\") - please send a bug report if this happens again"
execute_hook "after_resume_container"
log_trace "<-- Resume container"
reset_trap
return 0
fi
if [[ "$CT_STATUS_MOUNT" != "mounted" ]] || [[ "$CT_STATUS_RUNNING" != "running" ]] || [[ "$CT_STATUS_SUSPENDED" != "suspended" ]]; then
exit_critical "Unexpected container state (ctid: $TASK_CTID - status: \"$CT_STATUS\" - expected: \"exist mounted running suspended\") - please send a bug report"
return
fi
# ---
execute_hook "before_resume_container"
fi
local TASK_INIT="$($CMD_DATE "+%s%1N")"
set +o errexit; trap - ERR
if [[ "$CONFIG_TYPE" != "Virtuozzo" ]]; then
if [[ "$ARGS_VZCTL" == "--quiet" ]]; then
$CMD_VZCTL chkpnt "$TASK_CTID" --resume 2>> "$LOG_FILE" > "/dev/null"
else
$CMD_VZCTL $ARGS_VZCTL chkpnt "$TASK_CTID" --resume 2>&1 | $CMD_TEE -a "$LOG_FILE"
fi
else
log_info2 "$CONFIG_TYPE ${CONFIG_XVER:-Legacy} detected - unable to resume container (operation not supported)"
fi
local ERROR_CODE="$?"
set_trap "Unable to resume container"
local TASK_STOP="$($CMD_DATE "+%s%1N")"
# Note: mutable
DOWN_STOP="$TASK_STOP"
# --- Check error code
if [[ "$ERROR_CODE" -ne 0 ]]; then
check_container_status
exit_error "Unable to resume container (ctid: $TASK_CTID - status: \"$CT_STATUS\" - error: \"$ERROR_CODE\") - aborting"
return
fi
# ---
if [[ "$CONFIG_TYPE" != "Virtuozzo" ]]; then
if [[ "$(( OPTION_OPTIMIZE & 2 ))" -eq 2 ]]; then
log_info2 "Optimized mode ( 2) - container is resuming - waiting 5 seconds"
task_pause 5 force
fi
log_info2 "Container has been resumed - use \"$CMD_VZCTL chkpnt $TASK_CTID --suspend\" to suspend it manually"
execute_hook "after_resume_container"
fi
fi
# ---
local FUNC_STOP="$($CMD_DATE "+%s%1N")"
compute_time "$TASK_INIT" "$TASK_STOP"
log_time "$DISPLAY_STAT_TIME_1ST - Resume container (ctid: $TASK_CTID)"
log_trace "<-- Resume container"
reset_trap
}
# ---
function mount_container {
set_trap "Unable to mount container"
log_trace "--> Mount container"
local FUNC_INIT="$($CMD_DATE "+%s%1N")"
# --- task_restore
if [[ -n "${TARGET_CTID:-}" ]] && [[ "$TARGET_CTID" != "$BACKUP_CTID" ]]; then
log_debug "Switching backup/target CTID (backup: $BACKUP_CTID - target: $TARGET_CTID)"
local TASK_CTID="$TARGET_CTID"
else
local TASK_CTID="$BACKUP_CTID"
fi
# --- Check status
check_container_status
if [[ "$CT_STATUS_EXIST" != "exist" ]]; then
if [[ "$OPTION_DRYRUN" -eq 1 ]] && [[ "$CONFIG_TASK" == "restore" ]]; then
log_debug "Container not found (ctid: $TASK_CTID - status: \"$CT_STATUS\") - ignoring (dry-run mode)"
else
exit_critical "Container not found (ctid: $TASK_CTID - status: \"$CT_STATUS\") - please send a bug report"
return
fi
fi
# --- Mount container
if [[ "$OPTION_DRYRUN" -eq 1 ]]; then
execute_hook "before_mount_container"
local TASK_INIT="$($CMD_DATE "+%s%1N")"
dry_run "$CMD_VZCTL $ARGS_VZCTL mount \"$TASK_CTID\""
local TASK_STOP="$($CMD_DATE "+%s%1N")"
log_debug "Container has been mounted (dry-run mode)"
execute_hook "after_mount_container"
else
if [[ "$CT_STATUS_MOUNT" == "mounted" ]] && [[ "$CT_STATUS_RUNNING" == "down" ]] && [[ "$CT_STATUS_SUSPENDED" == "" ]]; then
# Note: ALL (mount_container: fallback)
log_warning "Container already mounted (ctid: $TASK_CTID - status: \"$CT_STATUS\") - please send a bug report if this happens again"
execute_hook "after_mount_container"
log_trace "<-- Mount container"
reset_trap
return 0
fi
if [[ "$CT_STATUS_MOUNT" != "unmounted" ]] || [[ "$CT_STATUS_RUNNING" != "down" ]] || [[ "$CT_STATUS_SUSPENDED" != "" ]]; then
exit_critical "Unexpected container state (ctid: $TASK_CTID - status: \"$CT_STATUS\" - expected: \"exist unmounted down\") - please send a bug report"
return
fi
# ---
execute_hook "before_mount_container"
local TASK_INIT="$($CMD_DATE "+%s%1N")"
set +o errexit; trap - ERR
if [[ "$ARGS_VZCTL" == "--quiet" ]]; then
$CMD_VZCTL mount "$TASK_CTID" 2>> "$LOG_FILE" > "/dev/null"
else
$CMD_VZCTL $ARGS_VZCTL mount "$TASK_CTID" 2>&1 | $CMD_TEE -a "$LOG_FILE"
fi
local ERROR_CODE="$?"
set_trap "Unable to mount container"
local TASK_STOP="$($CMD_DATE "+%s%1N")"
# --- Check error code
if [[ "$ERROR_CODE" -ne 0 ]]; then
check_container_status
exit_error "Unable to mount container (ctid: $TASK_CTID - status: \"$CT_STATUS\" - error: \"$ERROR_CODE\") - aborting"
return
fi
# ---
log_info2 "Container has been mounted - use \"$CMD_VZCTL umount $TASK_CTID\" to unmount it manually"
execute_hook "after_mount_container"
fi
# ---
local FUNC_STOP="$($CMD_DATE "+%s%1N")"
compute_time "$TASK_INIT" "$TASK_STOP"
log_time "$DISPLAY_STAT_TIME_1ST - Mount container (ctid: $TASK_CTID)"
log_trace "<-- Mount container"
reset_trap
}
# ---
function unmount_container {
set_trap "Unable to unmount container"
log_trace "--> Unmount container"
local FUNC_INIT="$($CMD_DATE "+%s%1N")"
# --- task_restore
if [[ -n "${TARGET_CTID:-}" ]] && [[ "$TARGET_CTID" != "$BACKUP_CTID" ]]; then
log_debug "Switching backup/target CTID (backup: $BACKUP_CTID - target: $TARGET_CTID)"
local TASK_CTID="$TARGET_CTID"
else
local TASK_CTID="$BACKUP_CTID"
fi
# --- Check status
check_container_status
if [[ "$CT_STATUS_EXIST" != "exist" ]]; then
if [[ "$OPTION_DRYRUN" -eq 1 ]] && [[ "$CONFIG_TASK" == "restore" ]]; then
log_debug "Container not found (ctid: $TASK_CTID - status: \"$CT_STATUS\") - ignoring (dry-run mode)"
else
exit_critical "Container not found (ctid: $TASK_CTID - status: \"$CT_STATUS\") - please send a bug report"
return
fi
fi
# --- Unmount container
if [[ "$OPTION_DRYRUN" -eq 1 ]]; then
execute_hook "before_unmount_container"
local TASK_INIT="$($CMD_DATE "+%s%1N")"
dry_run "$CMD_VZCTL $ARGS_VZCTL umount \"$TASK_CTID\""
local TASK_STOP="$($CMD_DATE "+%s%1N")"
log_debug "Container has been unmounted (dry-run mode)"
execute_hook "after_unmount_container"
else
if [[ "$CT_STATUS_MOUNT" == "unmounted" ]] && [[ "$CT_STATUS_RUNNING" == "down" ]] && [[ "$CT_STATUS_SUSPENDED" == "" ]]; then
# Note: ALL (unmount_container: fallback)
log_warning "Container already unmounted (ctid: $TASK_CTID - status: \"$CT_STATUS\") - please send a bug report if this happens again"
execute_hook "after_unmount_container"
log_trace "<-- Unmount container"
reset_trap
return 0
fi
if [[ "$CT_STATUS_MOUNT" != "mounted" ]] || [[ "$CT_STATUS_RUNNING" != "down" ]] || [[ "$CT_STATUS_SUSPENDED" != "" ]]; then
exit_critical "Unexpected container state (ctid: $TASK_CTID - status: \"$CT_STATUS\" - expected: \"exist mounted down\") - please send a bug report"
return
fi
# ---
execute_hook "before_unmount_container"
local TASK_INIT="$($CMD_DATE "+%s%1N")"
set +o errexit; trap - ERR
if [[ "$ARGS_VZCTL" == "--quiet" ]]; then
$CMD_VZCTL umount "$TASK_CTID" 2>> "$LOG_FILE" > "/dev/null"
else
$CMD_VZCTL $ARGS_VZCTL umount "$TASK_CTID" 2>&1 | $CMD_TEE -a "$LOG_FILE"
fi
local ERROR_CODE="$?"
set_trap "Unable to unmount container"
local TASK_STOP="$($CMD_DATE "+%s%1N")"
# --- Check error code
if [[ "$ERROR_CODE" -ne 0 ]]; then
check_container_status
exit_error "Unable to unmount container (ctid: $TASK_CTID - status: \"$CT_STATUS\" - error: \"$ERROR_CODE\") - aborting"
return
fi
# ---
log_info2 "Container has been unmounted - use \"$CMD_VZCTL mount $TASK_CTID\" to mount it manually"
execute_hook "after_unmount_container"
fi
# ---
local FUNC_STOP="$($CMD_DATE "+%s%1N")"
compute_time "$TASK_INIT" "$TASK_STOP"
log_time "$DISPLAY_STAT_TIME_1ST - Unmount container (ctid: $TASK_CTID)"
log_trace "<-- Unmount container"
reset_trap
}
# ---
function execute_hook {
set_trap "Unable to execute hook"
log_trace "--> Execute hook"
local FUNC_INIT="$($CMD_DATE "+%s%1N")"
# --- Check args
if [[ "$CONFIG_TASK" == "$CALLER_TASK" ]]; then
local HOOK_NAME="$1"
local HOOK_TYPE="batch"
local HOOK_TASK="$BATCH_TASK"
local HOOK_CTID="$BATCH_CTID"
else
local HOOK_NAME="$1"
local HOOK_TYPE="task"
local HOOK_TASK="$CONFIG_TASK"
local HOOK_CTID="$BACKUP_CTID"
fi
if [[ -n "${TARGET_CTID:-}" ]] && [[ "$TARGET_CTID" != "$HOOK_CTID" ]]; then
log_debug "Switching backup/target CTID (backup: $HOOK_CTID - target: $TARGET_CTID)"
local HOOK_CTID="$TARGET_CTID"
fi
# ---
if [[ "$HOOK_NAME" != "start" ]] && [[ "$HOOK_NAME" != "failure" ]] && [[ "$HOOK_NAME" != "success" ]] && [[ "$HOOK_NAME" != "exit" ]] &&
[[ "$HOOK_NAME" != "before_create_container" ]] && [[ "$HOOK_NAME" != "after_create_container" ]] &&
[[ "$HOOK_NAME" != "before_mount_container" ]] && [[ "$HOOK_NAME" != "after_mount_container" ]] &&
[[ "$HOOK_NAME" != "before_start_container" ]] && [[ "$HOOK_NAME" != "after_start_container" ]] &&
[[ "$HOOK_NAME" != "before_suspend_container" ]] && [[ "$HOOK_NAME" != "after_suspend_container" ]] &&
[[ "$HOOK_NAME" != "before_resume_container" ]] && [[ "$HOOK_NAME" != "after_resume_container" ]] &&
[[ "$HOOK_NAME" != "before_stop_container" ]] && [[ "$HOOK_NAME" != "after_stop_container" ]] &&
[[ "$HOOK_NAME" != "before_unmount_container" ]] && [[ "$HOOK_NAME" != "after_unmount_container" ]]; then
exit_critical "Invalid hook name (name: \"$HOOK_NAME\" - type: \"$HOOK_TYPE\" - task: \"$HOOK_TASK\" - ctid: $HOOK_CTID) - please send a bug report"
return
fi
if [[ "$HOOK_TYPE" != "batch" ]] && [[ "$HOOK_TYPE" != "task" ]]; then
exit_critical "Invalid hook type (name: \"$HOOK_NAME\" - type: \"$HOOK_TYPE\" - task: \"$HOOK_TASK\" - ctid: $HOOK_CTID) - please send a bug report"
return
fi
if [[ "$HOOK_TASK" != "backup" ]] && [[ "$HOOK_TASK" != "delete" ]] && [[ "$HOOK_TASK" != "destroy" ]] && [[ "$HOOK_TASK" != "download" ]] && [[ "$HOOK_TASK" != "upload" ]] && [[ "$HOOK_TASK" != "restore" ]] && [[ "$HOOK_TASK" != "update" ]]; then
exit_critical "Invalid hook task (name: \"$HOOK_NAME\" - type: \"$HOOK_TYPE\" - task: \"$HOOK_TASK\" - ctid: $HOOK_CTID) - please send a bug report"
return
fi
if [[ "$HOOK_CTID" != "all" ]] && [[ "$HOOK_CTID" == *[!0-9]* ]] && [[ "${#HOOK_CTID}" -ne 36 ]] && [[ "$HOOK_CTID" == *[!0-9a-f\-]* ]]; then
exit_critical "Invalid hook ctid (name: \"$HOOK_NAME\" - type: \"$HOOK_TYPE\" - task: \"$HOOK_TASK\" - ctid: $HOOK_CTID) - please send a bug report"
return
fi
if [[ -z "${PREVIOUS_HOOK:-}" ]] && [[ "$HOOK_NAME" != "start" ]]; then
exit_critical "Invalid hook call (name: \"$HOOK_NAME\" - type: \"$HOOK_TYPE\" - task: \"$HOOK_TASK\" - ctid: $HOOK_CTID) - please send a bug report"
return
fi
# ---
local TASK_INIT="$($CMD_DATE "+%s%1N")"
local HOOK_LIST="hook-${HOOK_NAME}-${HOOK_TYPE}-${HOOK_TASK}-${HOOK_CTID} hook-${HOOK_NAME}-${HOOK_TYPE}-${HOOK_TASK} hook-${HOOK_NAME}-${HOOK_TYPE} hook-${HOOK_NAME}"
local HOOK_NAME=""
for HOOK_NAME in $HOOK_LIST; do
if [[ ! -f "$CONFIG_PATH/etc/hooks/$HOOK_NAME" ]]; then
log_debug "Hook file not found (name: \"$HOOK_NAME\") - skipping"
continue
elif [[ ! -h "$CONFIG_PATH/etc/hooks/$HOOK_NAME" ]]; then
# Note: ALL (execute_hook: fallback)
log_warning "Hook file is not a symbolic link (name: \"$HOOK_NAME\") - ignoring (please, use a symlink and try again)"
continue
elif [[ ! -x "$CONFIG_PATH/etc/hooks/$HOOK_NAME" ]]; then
# Note: ALL (execute_hook: fallback)
log_warning "Hook file is not executable (name: \"$HOOK_NAME\") - ignoring (please, set correct permissions and try again)"
continue
fi
# ---
local HOOK_FILE="$($CMD_READLINK -e -n "$CONFIG_PATH/etc/hooks/$HOOK_NAME")"
local HOOK_FILE="${HOOK_FILE##*/}"
if [[ -z "$HOOK_FILE" ]]; then
exit_critical "Hook file not found (name: \"$HOOK_NAME\" - file: \"$HOOK_FILE\") - please send a bug report"
return
elif [[ "$HOOK_FILE" =~ ^OVZDB_ ]]; then
log_debug "Hook exec (name: \"$HOOK_NAME\" - file: \"$HOOK_FILE\")"
else
log_info "Hook exec (name: \"$HOOK_NAME\" - file: \"$HOOK_FILE\")"
fi
# ---
if [[ "$HOOK_FILE" == "OVZDB_DO_NOTHING" ]]; then
continue
fi
# ---
if [[ "$HOOK_TASK" == "backup" ]]; then
local HOOK_BACKUP_MODE="${BACKUP_MODE:-undefined}"
local HOOK_BACKUP_DATE="${BACKUP_DATE:-undefined}"
elif [[ "$HOOK_TASK" == "delete" ]]; then
local HOOK_BACKUP_MIN_DATE="${BACKUP_MIN_DATE:-undefined}"
local HOOK_BACKUP_MAX_DATE="${BACKUP_MAX_DATE:-undefined}"
local HOOK_BACKUP_STRG="${BACKUP_STRG:-undefined}"
elif [[ "$HOOK_TASK" == "destroy" ]]; then
local HOOK_DELETE_MODE="${DELETE_MODE:-undefined}"
local HOOK_BACKUP_STRG="${BACKUP_STRG:-undefined}"
elif [[ "$HOOK_TASK" == "download" ]]; then
local HOOK_BACKUP_MIN_DATE="${BACKUP_MIN_DATE:-undefined}"
local HOOK_BACKUP_MAX_DATE="${BACKUP_MAX_DATE:-undefined}"
elif [[ "$HOOK_TASK" == "upload" ]]; then
local HOOK_BACKUP_MIN_DATE="${BACKUP_MIN_DATE:-undefined}"
local HOOK_BACKUP_MAX_DATE="${BACKUP_MAX_DATE:-undefined}"
elif [[ "$HOOK_TASK" == "restore" ]]; then
local HOOK_BACKUP_MODE="${BACKUP_MODE:-undefined}"
local HOOK_BACKUP_DATE="${BACKUP_DATE:-undefined}"
local HOOK_TARGET_CTID="${TARGET_CTID:-undefined}"
local HOOK_TARGET_TYPE="${TARGET_TYPE:-undefined}"
elif [[ "$HOOK_TASK" == "update" ]]; then
local HOOK_UPDATE_MODE="${UPDATE_MODE:-undefined}"
local HOOK_UPDATE_TYPE="${UPDATE_TYPE:-undefined}"
fi
# ---
if [[ "$OPTION_DRYRUN" -eq 1 ]]; then
if [[ "$HOOK_TASK" == "backup" ]]; then
dry_run "$CONFIG_PATH/etc/hooks/$HOOK_NAME \"$LOG_FILE\" \"$CONFIG_HOST\" \"$CONFIG_PATH\" \"$CONFIG_TASK\" \"$CONFIG_DATE\" \"$CONFIG_FILE\" \"$HOOK_NAME\" \"$HOOK_TYPE\" \"$HOOK_TASK\" \"$HOOK_CTID\" \"$HOOK_BACKUP_MODE\" \"$HOOK_BACKUP_DATE\""
elif [[ "$HOOK_TASK" == "delete" ]]; then
dry_run "$CONFIG_PATH/etc/hooks/$HOOK_NAME \"$LOG_FILE\" \"$CONFIG_HOST\" \"$CONFIG_PATH\" \"$CONFIG_TASK\" \"$CONFIG_DATE\" \"$CONFIG_FILE\" \"$HOOK_NAME\" \"$HOOK_TYPE\" \"$HOOK_TASK\" \"$HOOK_CTID\" \"$HOOK_BACKUP_MIN_DATE\" \"$HOOK_BACKUP_MAX_DATE\" \"$HOOK_BACKUP_STRG\""
elif [[ "$HOOK_TASK" == "destroy" ]]; then
dry_run "$CONFIG_PATH/etc/hooks/$HOOK_NAME \"$LOG_FILE\" \"$CONFIG_HOST\" \"$CONFIG_PATH\" \"$CONFIG_TASK\" \"$CONFIG_DATE\" \"$CONFIG_FILE\" \"$HOOK_NAME\" \"$HOOK_TYPE\" \"$HOOK_TASK\" \"$HOOK_CTID\" \"$HOOK_DELETE_MODE\" \"$HOOK_BACKUP_STRG\""
elif [[ "$HOOK_TASK" == "download" ]]; then
dry_run "$CONFIG_PATH/etc/hooks/$HOOK_NAME \"$LOG_FILE\" \"$CONFIG_HOST\" \"$CONFIG_PATH\" \"$CONFIG_TASK\" \"$CONFIG_DATE\" \"$CONFIG_FILE\" \"$HOOK_NAME\" \"$HOOK_TYPE\" \"$HOOK_TASK\" \"$HOOK_CTID\" \"$HOOK_BACKUP_MIN_DATE\" \"$HOOK_BACKUP_MAX_DATE\""
elif [[ "$HOOK_TASK" == "upload" ]]; then
dry_run "$CONFIG_PATH/etc/hooks/$HOOK_NAME \"$LOG_FILE\" \"$CONFIG_HOST\" \"$CONFIG_PATH\" \"$CONFIG_TASK\" \"$CONFIG_DATE\" \"$CONFIG_FILE\" \"$HOOK_NAME\" \"$HOOK_TYPE\" \"$HOOK_TASK\" \"$HOOK_CTID\" \"$HOOK_BACKUP_MIN_DATE\" \"$HOOK_BACKUP_MAX_DATE\""
elif [[ "$HOOK_TASK" == "restore" ]]; then
dry_run "$CONFIG_PATH/etc/hooks/$HOOK_NAME \"$LOG_FILE\" \"$CONFIG_HOST\" \"$CONFIG_PATH\" \"$CONFIG_TASK\" \"$CONFIG_DATE\" \"$CONFIG_FILE\" \"$HOOK_NAME\" \"$HOOK_TYPE\" \"$HOOK_TASK\" \"$HOOK_CTID\" \"$HOOK_BACKUP_MODE\" \"$HOOK_BACKUP_DATE\" \"$HOOK_TARGET_CTID\" \"$HOOK_TARGET_TYPE\""
elif [[ "$HOOK_TASK" == "update" ]]; then
dry_run "$CONFIG_PATH/etc/hooks/$HOOK_NAME \"$LOG_FILE\" \"$CONFIG_HOST\" \"$CONFIG_PATH\" \"$CONFIG_TASK\" \"$CONFIG_DATE\" \"$CONFIG_FILE\" \"$HOOK_NAME\" \"$HOOK_TYPE\" \"$HOOK_TASK\" \"$HOOK_CTID\" \"$HOOK_UPDATE_MODE\" \"$HOOK_UPDATE_TYPE\""
fi
else
set +o errexit; trap - ERR
if [[ "$HOOK_TASK" == "backup" ]]; then
$CONFIG_PATH/etc/hooks/$HOOK_NAME "$LOG_FILE" "$CONFIG_HOST" "$CONFIG_PATH" "$CONFIG_TASK" "$CONFIG_DATE" "$CONFIG_FILE" "$HOOK_NAME" "$HOOK_TYPE" "$HOOK_TASK" "$HOOK_CTID" "$HOOK_BACKUP_MODE" "$HOOK_BACKUP_DATE" 2>&1 | $CMD_TEE -a "$LOG_FILE"
elif [[ "$HOOK_TASK" == "delete" ]]; then
$CONFIG_PATH/etc/hooks/$HOOK_NAME "$LOG_FILE" "$CONFIG_HOST" "$CONFIG_PATH" "$CONFIG_TASK" "$CONFIG_DATE" "$CONFIG_FILE" "$HOOK_NAME" "$HOOK_TYPE" "$HOOK_TASK" "$HOOK_CTID" "$HOOK_BACKUP_MIN_DATE" "$HOOK_BACKUP_MAX_DATE" "$HOOK_BACKUP_STRG" 2>&1 | $CMD_TEE -a "$LOG_FILE"
elif [[ "$HOOK_TASK" == "destroy" ]]; then
$CONFIG_PATH/etc/hooks/$HOOK_NAME "$LOG_FILE" "$CONFIG_HOST" "$CONFIG_PATH" "$CONFIG_TASK" "$CONFIG_DATE" "$CONFIG_FILE" "$HOOK_NAME" "$HOOK_TYPE" "$HOOK_TASK" "$HOOK_CTID" "$HOOK_DELETE_MODE" "$HOOK_BACKUP_STRG" 2>&1 | $CMD_TEE -a "$LOG_FILE"
elif [[ "$HOOK_TASK" == "download" ]]; then
$CONFIG_PATH/etc/hooks/$HOOK_NAME "$LOG_FILE" "$CONFIG_HOST" "$CONFIG_PATH" "$CONFIG_TASK" "$CONFIG_DATE" "$CONFIG_FILE" "$HOOK_NAME" "$HOOK_TYPE" "$HOOK_TASK" "$HOOK_CTID" "$HOOK_BACKUP_MIN_DATE" "$HOOK_BACKUP_MAX_DATE" 2>&1 | $CMD_TEE -a "$LOG_FILE"
elif [[ "$HOOK_TASK" == "upload" ]]; then
$CONFIG_PATH/etc/hooks/$HOOK_NAME "$LOG_FILE" "$CONFIG_HOST" "$CONFIG_PATH" "$CONFIG_TASK" "$CONFIG_DATE" "$CONFIG_FILE" "$HOOK_NAME" "$HOOK_TYPE" "$HOOK_TASK" "$HOOK_CTID" "$HOOK_BACKUP_MIN_DATE" "$HOOK_BACKUP_MAX_DATE" 2>&1 | $CMD_TEE -a "$LOG_FILE"
elif [[ "$HOOK_TASK" == "restore" ]]; then
$CONFIG_PATH/etc/hooks/$HOOK_NAME "$LOG_FILE" "$CONFIG_HOST" "$CONFIG_PATH" "$CONFIG_TASK" "$CONFIG_DATE" "$CONFIG_FILE" "$HOOK_NAME" "$HOOK_TYPE" "$HOOK_TASK" "$HOOK_CTID" "$HOOK_BACKUP_MODE" "$HOOK_BACKUP_DATE" "$HOOK_TARGET_CTID" "$HOOK_TARGET_TYPE" 2>&1 | $CMD_TEE -a "$LOG_FILE"
elif [[ "$HOOK_TASK" == "update" ]]; then
$CONFIG_PATH/etc/hooks/$HOOK_NAME "$LOG_FILE" "$CONFIG_HOST" "$CONFIG_PATH" "$CONFIG_TASK" "$CONFIG_DATE" "$CONFIG_FILE" "$HOOK_NAME" "$HOOK_TYPE" "$HOOK_TASK" "$HOOK_CTID" "$HOOK_UPDATE_MODE" "$HOOK_UPDATE_TYPE" 2>&1 | $CMD_TEE -a "$LOG_FILE"
fi
local ERROR_CODE="$?"
set_trap "Unable to execute hook"
# --- Check error code
if [[ "$ERROR_CODE" -ne 0 ]]; then
if [[ "$HOOK_NAME" == "start" ]]; then
exit_error "Hook failure (name: \"$HOOK_NAME\" - file: \"$HOOK_FILE\" - error: \"$ERROR_CODE\") - aborting (please, correct the error and try again)"
return
fi
# Note: ALL (execute_hook: fallback)
log_warning "Hook failure (name: \"$HOOK_NAME\" - file: \"$HOOK_FILE\" - error: \"$ERROR_CODE\") - ignoring (please, correct the error and try again)"
continue
fi
if [[ "$HOOK_FILE" == "OVZDB_DO_NOTHING" ]] || [[ "$HOOK_FILE" == "OVZDB_SEND_DEBUG_REPORT" ]] || [[ "$HOOK_FILE" == "OVZDB_SEND_ERROR_REPORT" ]]; then
log_trace "Hook success (name: \"$HOOK_NAME\" - file: \"$HOOK_FILE\")"
else
log_debug "Hook success (name: \"$HOOK_NAME\" - file: \"$HOOK_FILE\")"
fi
fi
done
local TASK_STOP="$($CMD_DATE "+%s%1N")"
# ---
# Note: mutable
PREVIOUS_HOOK="$HOOK_NAME"
# ---
local FUNC_STOP="$($CMD_DATE "+%s%1N")"
compute_time "$TASK_INIT" "$TASK_STOP"
log_time "$DISPLAY_STAT_TIME_1ST - Execute hook \"${HOOK_NAME#hook-}\""
log_trace "<-- Execute hook"
reset_trap
}
# ---
function display_version {
$CMD_ECHO "#----------------------------------------------------#"
$CMD_ECHO "| openvz-diff-backups 1.0.1.12 (stable) (2020-06-06) |"
$CMD_ECHO "#----------------------------------------------------#"
$CMD_ECHO ""
}
# ---
function display_options {
# ### Debug
#
# backtrace:
# -b : display bash backtrace (for debug purpose only)
#
# xtrace:
# -x : enable bash xtrace mode (for debug purpose only)
$CMD_ECHO "
# --- Options:
### All tasks
config:
-c <file> : use alternate configuration file
(ex: $CALLER_TASK backup 101 auto -c /etc/$CALLER_TASK.conf)
dry-run:
-d : perform a trial run that doesn't make any changes
(ex: $CALLER_TASK backup 101 auto -d)
log-level:
-l <0-9> : set log level (screen/stdout), default is 7 (info),
(ex: $CALLER_TASK backup 101 auto -l6)
0 -> disable logging (not recommended)
1 -> emerg, 2 -> alert, 3 -> critical,
4 -> error, 5 -> warning, 6 -> notice,
7 -> info, 8 -> info2, 9 -> time,
10 -> debug, 11 -> trace
(log files always record activity from level 1 up to level 9)
pause:
-p : wait 3 seconds before each important step
(ex: $CALLER_TASK backup 101 auto -p)
quiet:
-q : minimize splash screen
-qq : print only one status line for each container processed
-qqq : 10000 foot view, display only useful information
(ex: $CALLER_TASK backup 101 auto -q)
turbo:
-t : disable all throttling parameters, task will operate at full speed with idle priority (I/O and bandwidth intensive)
-tt : same as above but with a normal priority ($CMD_IONICE -c 3 -n 0 & $CMD_NICE -n 0)
-ttt : same as above but with a higher priority ($CMD_IONICE -c 2 -n 0 & $CMD_NICE -n -10)
(ex: $CALLER_TASK backup 101 auto -t)
unlock:
-u : override locking mechanisms (will unlock container and master/remote storage, use with caution)
(ex: $CALLER_TASK backup 101 auto -u)
verbose:
-v : verbose (display file changes)
-vv : very verbose (ssh, rsync, vzctl, lvm2, ploop, bzip2, ...)
-vvv : very very verbose (totally insane output, for debug purpose only)
(ex: $CALLER_TASK backup 101 auto -v)
wait:
-w <date> : wait until <date> before starting task (deferred start: please use a near future date)
(ex: $CALLER_TASK backup 101 auto -w $($CMD_DATE -d "@$(( $($CMD_DATE -d "now 3600 seconds" "+%s") / 3600 * 3600 ))" "+%Y-%m-%d_%H-%M-%S"))
-w <n> : wait until current time modulo <n> seconds equals zero before starting task
(ex: $CALLER_TASK backup 101 auto -w 300
will be translated to
$CALLER_TASK backup 101 auto -w $($CMD_DATE -d "@$(( $($CMD_DATE -d "now 300 seconds" "+%s") / 300 * 300 ))" "+%Y-%m-%d_%H-%M-%S"))
### Backup task
force:
-f : force backup task to save containers using selected backup mode (will ignore \"CONTAINERS_FORCE_[COLD|HOLD|LIVE]_BACKUP\" parameters)
(ex: $CALLER_TASK backup 101 cold -f)
hostname:
-h <fqdn> : filter backups to create by hostname (won't reuse previous backups from other hosts)
(ex: $CALLER_TASK backup 101 auto -h $($CMD_HOSTNAME -f))
keep:
-k <n> : preserve backup for at least <n> days (\"sticky backups\" can survive delete task, unless \"force\" option is used)
(ex: $CALLER_TASK backup 101 auto -k 30)
optimize:
-o <n> : enable one or multiple optimisations based on their numbers (ex: scan inodes (1) + use bzip2 (4) = "-o 5")
1 : scan inodes inside container before syncing them (should decrease server load and reduce I/O spikes - your mileage may vary)
2 : sleep 5/10 seconds after resuming/starting container and 5 seconds before unmounting/deleting LVM2/ploop snapshot
4 : compress memory dump and other data with bzip2 (mono-core) instead of pbzip2 (multi-core)
8 : pre-sync container data before doing a backup (should decrease suspend time when LVM2/ploop snapshots are not available)
16 : detect and use "nocache" command (on source and master hosts) to preserve legit kernel page cache when creating a backup
32 : disable backup size calculation when creating a backup (will decrease VFS cache pressure, useful if master host is low on RAM)
(ex: $CALLER_TASK backup 101 auto -o 20)
resync:
-r : compare content of every path/file/other when creating an incremental backup (compute 128-bit checksum)
-rr : backup data as a full backup instead of an incremental one (delta-transfer algorithm)
(ex: $CALLER_TASK backup 101 auto -r)
### Delete task
force:
-f : force delete task to erase all backups found in date range (will delete \"sticky backups\" before their real expiration date)
(ex: $CALLER_TASK delete 101 7- -f)
hostname:
-h <fqdn> : filter backups to delete by hostname (won't delete backups created by other hosts)
(ex: $CALLER_TASK delete 101 7- -h $($CMD_HOSTNAME -f))
max-count:
-m <n> : set maximum number of backups to delete (for each container)
(ex: $CALLER_TASK delete 101 7- -m 1)
### Destroy task
keep:
-k <n> : set maximum number of inactive days before destroying storage
(ex: $CALLER_TASK destroy 101 auto -k 7)
max-count:
-m <n> : set maximum number of remaining backups before destroying storage
(ex: $CALLER_TASK destroy 101 auto -m 0)
### Download task
hostname:
-h <fqdn> : filter backups to download by hostname (won't download backups created by other hosts)
(ex: $CALLER_TASK download 101 0- -h $($CMD_HOSTNAME -f))
max-count:
-m <n> : set maximum number of backups to download (for each container)
(ex: $CALLER_TASK download 101 0- -m 1
optimize:
-o <n> : enable one or multiple optimisations based on their numbers
1 : scan inodes inside backup before syncing them (should decrease server load and reduce I/O spikes - your mileage may vary)
16 : detect and use "nocache" command (on remote and master hosts) to preserve legit kernel page cache when downloading a backup
32 : disable backup size calculation when downloading a backup (will decrease VFS cache pressure, useful if master host is low on RAM)
(ex: $CALLER_TASK download 101 0- -o 48)
resync:
-r : compare content of every path/file/other when downloading an incremental backup (compute 128-bit checksum)
-rr : download data as a full backup instead of an incremental one (delta-transfer algorithm)
(ex: $CALLER_TASK download 101 0- -r)
### Restore task
force:
-f : force restore task to overwrite existing containers (will replace all data, unless container is running or mounted)
(ex: $CALLER_TASK restore 101 auto -f)
hostname:
-h <fqdn> : filter backups to restore by hostname (won't restore backups created by other hosts)
(ex: $CALLER_TASK restore 101 auto -h $($CMD_HOSTNAME -f))
resync:
-r : compare content of every path/file/other when restoring an incremental backup (compute 128-bit checksum)
-rr : restore data as a full backup instead of an incremental one (delta-transfer algorithm)
(ex: $CALLER_TASK download 101 auto -r)
storage:
-s <path> : set alternative private storage path
(ex: $CALLER_TASK restore 101 auto -s /home/storage/vz/private)
### Upload task
hostname:
-h <fqdn> : filter backups to upload by hostname (won't upload backups created by other hosts)
(ex: $CALLER_TASK upload 101 0- -h $($CMD_HOSTNAME -f))
max-count:
-m <n> : set maximum number of backups to upload (for each container)
(ex: $CALLER_TASK upload 101 0- -m 1
optimize:
-o <n> : enable one or multiple optimisations based on their numbers
1 : scan inodes inside backup before syncing them (should decrease server load and reduce I/O spikes - your mileage may vary)
16 : detect and use "nocache" command (on master and remote hosts) to preserve legit kernel page cache when uploading a backup
32 : disable backup size calculation when uploading a backup (will decrease VFS cache pressure, useful if remote host is low on RAM)
(ex: $CALLER_TASK upload 101 0- -o 48)
resync:
-r : compare content of every path/file/other when uploading an incremental backup (compute 128-bit checksum)
-rr : upload data as a full backup instead of an incremental one (delta-transfer algorithm)
(ex: $CALLER_TASK upload 101 0- -r)
### Update task
force:
-f : force update task to install a major release (1 - it might/could break backward compatibility, 2 - if already up to date, will reinstall latest release)
(ex: $CALLER_TASK update all auto -f)
"
}
# ---
# exit
# Note: ALL (bash: force shell buffering)
}
More information about the Users
mailing list