Add incremential option

README.md

@@ -4,8 +4,8 @@
 
 A Bash script to generate a tar.gz backup of a folder, with an option to automatically upload the backup file to a cloud service using [rclone](https://github.com/rclone/rclone).  
 List of cloud/storage providers currently supported by rclone can be found [here](https://github.com/rclone/rclone#storage-providers).
-<br>  
-Latest version: 1.2.0 ([changelog](https://github.com/MichaelYochpaz/Backup-Script/blob/master/changelog.md))
+
+Latest version: 1.3.0 ([changelog](changelog.md))
 
 ## Features
 * Generate a tar.gz backup file of a folder.
@@ -23,10 +23,16 @@ Latest version: 1.2.0 ([changelog](https://github.com/MichaelYochpaz/Backup-Scri
 Configuration can be set on the script file under the "Configuration" section,  
 or using command-line arguments (will override configuration that's set on on the script).
 ```
- Usage: backup [-n <name>] [-s <path>] [-e <pattern>]... [-u <path>] [-r] [-v] <path-to-backup>
+ Usage: backup [-n <name>] [-d <format>] [-l <number>] [-s <path>] [-e <pattern>]... [-u <path>] [-p <API key>] [-r] [-y] [-v] <path-to-backup>
 
  Options:
    -n <name>     Sets the tar.gz file name [default: "backup"]
+   -d <format>   Sets date command format [default: "%FT%H%M" ie: "2024-03-05T0701"]
+   -l <number>   Incremental backup mode: Adds 'level<number>' to the filename.
+                 If the number is greater than 0, an incremental backup is performed.
+                 Full backups are performed by default (and when a snar file does not exist), 
+                 but indicating a level will create a snar file that will store data 
+                 necessary to perform future backups.
    -s <path>     Path to which the generated backup file will be saved to [default: current working directory]
    -e <pattern>  Exclude a pattern (specific files / folders) from being backed up
    -u <path>     rclone path to which the backup file will be uploaded to (not providing one will skip the upload process)
@@ -42,6 +48,7 @@ or using command-line arguments (will override configuration that's set on on th
  backup "/home/user/important_stuff"
  backup -u "GDrive:/Backups" -r -y -p "XXXXXXXXXXXXXXXX" "/home/user/important_stuff/" 
  backup -n "important-stuff-backup" -s "/home/user/backups" -e "*.pdf" -e "important_stuff/dont_backup_this_folder" "/home/user/important_stuff/"
+ backup -l 0 -s /mnt/backup /home/michael
 ```
 
 ## FAQ
@@ -52,4 +59,4 @@ or using command-line arguments (will override configuration that's set on on th
 **A:** Go to your Pushbullet account's [settings page](https://www.pushbullet.com/#settings/account), and click the "Create Access Token" button.
 
 **Q:** I found a bug, or have an idea for a feature. How can I help?  
-**A:** Feel free to open an [issue](https://github.com/MichaelYochpaz/Backup-Script/issues) and post your issue / suggestion.
+**A:** Feel free to open an issue and post your issue / suggestion.

backup

@@ -1,8 +1,9 @@
 #!/bin/bash
 ## ------------------------------------
 ##  Made By: Michael Yochpaz (C) 2020
-##  https://github.com/MichaelYochpaz/Backup-Script
-##  Version: 1.2.0
+##  Contributor: Michael Soh (github:sohmc)
+##  https://github.com/sohmc/Backup-Script
+##  Version: 1.3.0
 ##  License: GPLv3
 ## ------------------------------------
 ## -------------- Usage ---------------
@@ -13,10 +14,16 @@
 ##  - In order to use the upload feature, rclone must be installed and have at least one cloud service setup in config.
 ## ------------------------------------
 ##
-##  Usage: backup [-n <name>] [-s <path>] [-e <pattern>]... [-u <path>] [-p <API key>] [-r] [-y] [-v] <path-to-backup>
+##  Usage: backup [-n <name>] [-d <format>] [-l <number>] [-s <path>] [-e <pattern>]... [-u <path>] [-p <API key>] [-r] [-y] [-v] <path-to-backup>
 ##
 ##  Options:
 ##    -n <name>     Sets the tar.gz file name [default: "backup"]
+##    -d <format>   Sets date command format [default: "%FT%H%M" ie: "2024-03-05T0701"]
+##    -l <number>   Incremental backup mode: Adds 'level<number>' to the filename.
+##                  If the number is greater than 0, an incremental backup is performed.
+##                  Full backups are performed by default (and when a snar file does not exist), 
+##                  but indicating a level will create a snar file that will store data 
+##                  necessary to perform future backups.
 ##    -s <path>     Path to which the generated backup file will be saved to [default: current working directory]
 ##    -e <pattern>  Exclude a pattern (specific files / folders) from being backed up
 ##    -u <path>     rclone path to which the backup file will be uploaded to (not providing one will skip the upload process)
@@ -44,13 +51,15 @@ REMOVE_LOCAL=false # Boolean - Remove local backup file after upload.
 SKIP_WARNINGS=false # Boolean - Skip warnings.
 VERBOSE=false # Boolean - Use verbose mode when running tar and rclone.
 PUSHBULLET_TITLE="Backup Script" # Pusbullet - Notification title.
+DATE_FORMAT='%Y%m%d-%H%M%S'
+INCREMENTAL_LEVEL=-1
 PUSHBULLET_MESSAGE_START="Backup script " # Pushbullet - Notification messages will start with this string, followed by one of the followings:
 PUSHBULLET_MESSAGE_FINISHED_SUCCESS="finished successfully." # Pusbullet - Message will be shown if the script ran successfully.
 PUSHBULLET_MESSAGE_FINISHED_ERRORS="finished with errors." # Pusbullet - Message will be shown if the script finished with errors.
 PUSHBULLET_MESSAGE_FAIL="failed." # Pusbullet - Message will be shown if the script has failed.
 ## ------------------------------------
 
-usage() { echo "Usage: $(basename $0) [-n <name>] [-s <path>] [-e <pattern>]... [-u <path>] [-p <API key>] [-r] [-y] [-v] <path-to-backup>
+usage() { echo "Usage: $(basename $0) [-n <name>] [-d <format>] [-l <number>] [-s <path>] [-e <pattern>]... [-u <path>] [-p <API key>] [-r] [-y] [-v] <path-to-backup>
 Use $(basename $0) -h for additional info."; exit 1; }
 
 # if there is a Pushbullet API key, send a Pushbullet notification. uses a parameter for message's body
@@ -65,11 +74,17 @@ C2='\033[1;31m' # ANSI - error - red
 C3='\033[0;36m' # ANSI - information - cyan
 C4='\033[0;33m' # ANSI - variables/paths - orange
 
-while getopts ":n:e:s:u:p:ryvh" arg; do
+while getopts ":n:d:l:e:s:u:p:ryvh" arg; do
   case $arg in
     n)
       BACKUP_NAME=${OPTARG}
       ;;
+    d)
+      DATE_FORMAT=${OPTARG}
+      ;;
+    l)
+      INCREMENTAL_LEVEL=${OPTARG}
+      ;;
     e)
       EXCLUDES+=("${OPTARG}")
       ;;
@@ -92,9 +107,15 @@ while getopts ":n:e:s:u:p:ryvh" arg; do
       VERBOSE=true
       ;;
     h)
-      echo " Usage: $(basename $0) [-n <name>] [-s <path>] [-e <pattern>]... [-u <path>] [-p <API key>] [-r] [-y] [-v] <path-to-backup>
+      echo " Usage: $(basename $0) [-n <name>] [-d <format>] [-l <number>] [-s <path>] [-e <pattern>]... [-u <path>] [-p <API key>] [-r] [-y] [-v] <path-to-backup>
  Options:
    -n <name>     Sets the tar.gz file name [default: "backup"]
+   -d <format>   Sets date command format [default: "%FT%H%M" ie: "2024-03-05T0701"]
+   -l <number>   Incremental backup mode: Adds 'level<number>' to the filename.
+                 If the number is greater than 0, an incremental backup is performed.
+                 Full backups are performed by default (and when a snar file does not exist), 
+                 but indicating a level will create a snar file that will store data 
+                 necessary to perform future backups.
    -s <path>     Path to which the generated backup file will be saved to [default: current working directory]
    -e <pattern>  Exclude a pattern (specific files / folders) from being backed up
    -u <path>     rclone path to which the backup file will be uploaded to (not providing one will skip the upload process)
@@ -149,19 +170,70 @@ if [ "$REMOVE_LOCAL" = true ] && [ -z "$RCLONE_FOLDER" ] && [] "$SKIP_WARNINGS"
   if [[ ! $REPLY =~ ^[Yy]$ ]]; then exit 0; fi
 fi
 
+# Create Kebabed folder name so that we can use it as a file name
+KEBAB_BACKUP_FOLDER=${BACKUP_FOLDER//\//-}
+# Remove dashes or dots if they are in the front of the kebab
+shopt -s extglob
+KEBAB_BACKUP_FOLDER="${KEBAB_BACKUP_FOLDER##+([.-])}"
+shopt -u extglob
+# if INCREMENTAL_LEVEL is GTE 0, enable incremental label
+INCREMENTAL_LABEL=""
+INCREMENTAL_FLAG=""
+if [ ${INCREMENTAL_LEVEL} -ge 0 ]; then
+  TAR_CACHE=$HOME/.cache/tar
+  SNAR_FILE=$TAR_CACHE/$KEBAB_BACKUP_FOLDER.snar
+  INCREMENTAL_LABEL="-level${INCREMENTAL_LEVEL}"
+  INCREMENTAL_FLAG="--listed-incremental=${SNAR_FILE}"
+
+  echo -e "$(date +"%Y/%m/%d %T") ${C3}Incremental level set to $INCREMENTAL_LEVEL"${C0}
+
+  if [ ! -d $TAR_CACHE ]; then 
+    echo -e "$(date +"%Y/%m/%d %T") ${C3}Creating ${C4}$TAR_CACHE ${C3} directory"${C0}
+    mkdir -p $TAR_CACHE
+  fi
+
+  if [ -f $SNAR_FILE ]; 
+    then echo -e "$(date +"%Y/%m/%d %T") ${C3}snar file ${C4}$SNAR_FILE ${C3}found"${C0}
+  else 
+    echo -e "$(date +"%Y/%m/%d %T") ${C3}snar file ${C4}$SNAR_FILE ${C3}not found.  It will be created and a full backup will be performed."${C0}
+  fi
+
+  if [ ${INCREMENTAL_LEVEL} -eq 0 ] && [ -f $SNAR_FILE ]; then
+    echo -e "$(date +"%Y/%m/%d %T") ${C3}Incremental Level set to 0 and snar file exists.  Deleting..."${C0}
+    rm $SNAR_FILE
+  fi
+fi
+
 # ----- tar -----
 echo -e "$(date +"%Y/%m/%d %T") ${C3}Generating tar.gz file"${C0}
-FILENAME="${BACKUP_NAME}-$(date +%FT%H%M).tar.gz"
+FILENAME="${BACKUP_NAME}-$(date +${DATE_FORMAT})${INCREMENTAL_LABEL}.tar.gz"
+echo -e "$(date +"%Y/%m/%d %T") ${C3}tar FILENAME: ${C4}${FILENAME}"${C0}
 # generate a temporary file to store EXCLUDES in
-EXCLUEDS_FILE=$(mktemp /tmp/backup.XXXXXX.txt) 
-if [ ! ${#EXCLUDES[@]} -eq 0 ]; then printf "%s\n" "${EXCLUDES[@]}" > "$EXCLUEDS_FILE"; fi
+EXCLUDES_FILE=$(mktemp /tmp/backup.XXXXXX.txt)
+if [ ! ${#EXCLUDES[@]} -eq 0 ]; then printf "%s\n" "${EXCLUDES[@]}" > "$EXCLUDES_FILE"; fi
+
+# Include default.exclude if it exists
+DEFAULT_EXCLUDE=$HOME/.config/tar/default.exclude 
+if [ -f $DEFAULT_EXCLUDE ]; then 
+  echo -e "$(date +"%Y/%m/%d %T") ${C3} Default exclude file found.  Adding to excludes..."${C0}
+  cat $DEFAULT_EXCLUDE >> "$EXCLUDES_FILE";
+fi
+# Include a {kebabbed}.exclude if it exists
+KEBAB_BACKUP_FOLDER_EXCLUDE=$HOME/.config/tar/$KEBAB_BACKUP_FOLDER.exclude
+if [ -f $KEBAB_BACKUP_FOLDER_EXCLUDE ]; then 
+  echo -e "$(date +"%Y/%m/%d %T") ${C3} Exclude file for ${KEBAB_BACKUP_FOLDER} found.  Adding to excludes..."${C0}
+  cat $KEBAB_BACKUP_FOLDER_EXCLUDE >> "$EXCLUDES_FILE"; 
+fi
+
 
 # run tar with or without '-v', according to VERBOSE value
-if [ "$VERBOSE" = true ]; then tar -cvzf "$SAVE_FOLDER/$FILENAME" -C "$BACKUP_FOLDER/.." -X $EXCLUEDS_FILE "$(basename "$BACKUP_FOLDER")"; \
-else tar -czf "$SAVE_FOLDER/$FILENAME" -C "$BACKUP_FOLDER/.." -X $EXCLUEDS_FILE "$(basename "$BACKUP_FOLDER")"; fi
+VERBOSE_FLAG="";
+if [ "$VERBOSE" = true ]; then VERBOSE_FLAG="-v"; fi
+
+tar -cz $VERBOSE_FLAG -f "$SAVE_FOLDER/$FILENAME" $INCREMENTAL_FLAG -C "$BACKUP_FOLDER/.." -X $EXCLUDES_FILE "$(basename "$BACKUP_FOLDER")";
 
 # remove temporary file
-rm $EXCLUEDS_FILE
+rm $EXCLUDES_FILE
 
 # tar commmand finished successfully (exit code = 0)
 if [ "$?" = 0 ]; then
@@ -179,7 +251,7 @@ fi
 if [ -n "$RCLONE_FOLDER" ]; then
   echo -e "$(date +"%Y/%m/%d %T") ${C3}Uploading backup file to ${C4}$RCLONE_FOLDER${C0}"
   # run rclone with or without '-v', according to VERBOSE value
-  if [ "$VERBOSE" = true ]; then rclone copy "$SAVE_FOLDER/$FILENAME" -v "$RCLONE_FOLDER"; else rclone copy "$SAVE_FOLDER/$FILENAME" "$RCLONE_FOLDER"; fi
+  rclone copy "$SAVE_FOLDER/$FILENAME" $VERBOSE_FLAG "$RCLONE_FOLDER"
   # rclone commmand finished successfully (exit code = 0)
   if [ "$?" = 0 ]; then
     echo -e "$(date +"%Y/%m/%d %T") ${C1}Backup file uploaded successfully${C0}"
@@ -214,4 +286,4 @@ else
   echo -e "$(date +"%Y/%m/%d %T") ${C2}Script finished with errors${C0}"
   pushbullet "$PUSHBULLET_MESSAGE_FINISHED_ERRORS"
   exit 1
-fi
+fi

changelog.md

@@ -1,6 +1,12 @@
 # Changelog
 All notable changes to the script will be documented here
 
+## 1.3.0 - [2024-12-23]
+* Added feature to set your own `date` format via `-d` argument, followed by a valid format
+* Added feature to do incremential backups via the `-l` argument followed by an integer zero or greater
+* Script will look for a default exclude list, which is provided within this repo that's been shamelessly copied from [Timeshift](https://github.com/linuxmint/timeshift/blob/07f5589ecd1ff07036fa4a813292a2b5f51baefd/src/Core/Main.vala#L538)
+* Relatedly, the script will look for an exclude list for your dataset.  Exclude files are stored in `$HOME/.config/tar/$DATASET_NAME.exclude`.  Dataset name is derived by replacing slashes in the dataset name with a dash, removing any dots or dashes in the beginning of the dataset name.  Best way to use this is to provide the absolute path for the dataset.
+
 ## 1.2.0 - [2020-06-18] 
 * Added Pushbullet notifications support using the `-p` argument
 * Added `-y` argument to skip warnings (which by default, require user input to continue)

default.exclude

@@ -0,0 +1,59 @@
+dev/*
+proc/*
+sys/*
+media/*
+mnt/*
+tmp/*
+run/*
+var/run/*
+var/lock/*
+var/lib/dhcpcd/*
+var/lib/docker/*
+var/lib/schroot/*
+lost+found
+timeshift/*
+timeshift-btrfs/*
+data/*
+DATA/*
+cdrom/*
+sdcard/*
+system/*
+etc/timeshift.json
+var/log/timeshift/*
+var/log/timeshift-btrfs/*
+swapfile
+snap/*
+root/.thumbnails
+root/.cache
+root/.dbus
+root/.gvfs
+root/.local/share/[Tt]rash
+home/*/.thumbnails
+home/*/.cache
+home/*/.dbus
+home/*/.gvfs
+home/*/.local/share/[Tt]rash
+home/*/.thumbnails
+home/*/.cache
+home/*/.dbus
+home/*/.gvfs
+home/*/.local/share/[Tt]rash
+root/.mozilla/firefox/*.default/Cache
+root/.mozilla/firefox/*.default/OfflineCache
+root/.opera/cache
+root/.kde/share/apps/kio_http/cache
+root/.kde/share/cache/http
+home/*/.mozilla/firefox/*.default/Cache
+home/*/.mozilla/firefox/*.default/OfflineCache
+home/*/.vscode-server
+home/*/.opera/cache
+home/*/.kde/share/apps/kio_http/cache
+home/*/.kde/share/cache/http
+var/cache/apt/archives/*
+var/cache/pacman/pkg/*
+var/cache/yum/*
+var/cache/dnf/*
+var/cache/eopkg/*
+var/cache/xbps/*
+var/cache/zypp/*
+var/cache/edb/*