Incremental Backup with TAR / simple FTP to another location and email status

This is my script for archiving incremental and full backup with TAR in Linux and then FTP that archive to another 'server' for security if server loses data.
This is not complete backup to make it possible to restore 'bare metal' for that case use Mondo Backup.

REQUIREMENTS:
The script requires GNU TAR which is capable of Incremental archiving

Optional:
The LFTP improved FTP client for Linux capable of auto retrying the transfer until finished successfully

After I spent some time discovering The BIG BANG of Universe and The Meaning of Life :lol I managed somehow to create a script to make some backup of files on server and TAR/GZIP it and then FTP the archive to another FTP server and finally email the results.
This script also measures time needed to complete it and deletes archive older than xx days (set in find -mtime +20) and makes incremental backup every weekday and then FULL BACKUP on Sundays (which suits me bcoz no heavy load).

This is the script I had written to work for ME (you will have to modify it for yourself, I hope you find what and where), since I put it in CRON making it run every day

Put the scripts (files) to some directory where you will be making backups to, I use

/usr/tmp/serverbackups

Files for TAR to include and exclude are in plain txt format and filenames listed each name in separate line (these paths will be included in TAR-GZIP archive):
file: including.txt:

/var/
/etc/
/home/

For excluding the files / directories the syntax is:
- var/tmp  <-- exclude directory matching 'var/tmp' in the name
- spool  <-- exclude files matching 'spool' in the name: e.g. 'spool1 spoolwhatever' also *_log* matches names including '_log'
- var/tmp/serverbackups  <-- exclude directory with backups in it so we don't archive ourselves when creating new archives - obviously !

file: excluding.txt:

var/tmp/serverbackups
proc
*_log*
var/tmp
var/lib/bluetooth
var/lib/cs
var/lib/dav
var/lib/dbus
var/lib/dhcpv6
var/lib/dovecot
var/lib/games
var/lib/rpm
var/lib/webalizer
var/lib/yum
var/log
var/run
var/www/manual
var/yp
var/lib/php/session
spool
var/cache
*zip
*gz
etc/rc*
home/httpd/manual
rpm

I'm using LFTP to make sure that FTP transaction runs complete since ftp didn't always finish, the script for transferring the BACKUP file:
lftpscript.sh:

#!/bin/sh
HOST='ftp.domain.com'
USER='ftpuser'
PASSWD='idontknow'
FILE=$(cat archivename.txt)

lftp -c "open $HOST && user $USER $PASSWD && cd FOLDER_NAME_FOR_STORING/backups/ && put $FILE" <<END_SCRIPT
# all in one command that connects to HOST=ftp.domain.com with Username/password
# and changes directory to whatever you need
# then transfers the file with the name of created archive in archivename.txt (ex. BACKUP-2009-12-15-Fri-01-15-01h.tgz)
bye
exit
END_SCRIPT
exit 0

Crontab running the script, this is located in PATH '/usr/local/bin':

PATH=/sbin:/bin:/usr/bin:/usr/sbin:/usr/local/bin

0 1 * * * archive.sh >/dev/null         # runs the script at 1.00AM every day

runs the script archive.sh which only changes the directory path to '/usr/tmp/serverbackups' where the real Backup script is located.
Then it runs the backup script.
file: archive.sh:

#!/bin/bash
cd /usr/tmp/server_backup
./backup.sh         # the real script to make backup

Now comes the 'real' script which handles the archiving with TAR and sends over FTP to another location.

file: backup.sh:

#!/bin/sh

# DELETE archive older than -mtime +'days'
find . -name 'BACKUP*.tgz' -mtime +20 -delete
find . -name 'stopwatch*' -mtime +2 -delete

start1=$(date +"%T h ( %s )")
start=$(date +%s)

# on SUNDAY make FULL backup
if [ $(date +"%a") = "Sun" ]; then
{
SNAPSHOTFILE="./usr-full"; # needed by TAR (GNU-TAR to be precise) which is used to compare for incremental backups
ARCHIVENAME=BACKUP-full-$(date +"%F-%a-%H-%M-%Sh").tgz; # self explaining
rm -f "./usr-full";
}

else
{
ARCHIVENAME=BACKUP-$(date +"%F-%a-%H-%M-%Sh").tgz;
# a name of a backup file: BACKUP-2009-12-15-Fri-01-15-01h.tgz
SNAPSHOTFILE="./usr-1";
cp "./usr-full" "./usr-1";
}
fi

echo $ARCHIVENAME >archivename.txt # need the name to FTP transfer so store it in file archivename.txt which doesn't change (used later in lftpscript.sh ) !
# creating text to send in email
echo "-----------------------" >stopwatch-$archivename.txt
echo "Backup of $ARCHIVENAME" >>stopwatch-$archivename.txt
echo "-----------------------" >>stopwatch-$archivename.txt
echo " " >>stopwatch-$archivename.txt # echo " " makes new line /CR or LF whatever it does
# I do not need this precise time { time tar -T including.txt -X excluding.txt -pczvRf $ARCHIVENAME; } 2>> stopwatch-$ARCHIVENAME.txt >/dev/null
{ tar -T including.txt -X excluding.txt -pczvR --listed-incremental=$SNAPSHOTFILE -f $ARCHIVENAME; } 2>> stopwatch-$ARCHIVENAME.txt >/dev/null

stopped1=$(date +"%T h ( %s )")
stopped=$(date +%s)
ftpstarted=$stopped

thetime=$(($stopped-$start)) # doing some math in shell that's why $()
ELAPSEDHRS=$((($thetime%86400/3600)))
ELAPSSEC=$(($thetime%3600))


echo " " >>stopwatch-$ARCHIVENAME.txt
echo -n "File Size (Byte-s) : " >>stopwatch-$ARCHIVENAME.txt
ls -al "$ARCHIVENAME" | cut -f 5 -d ' ' >>stopwatch-$ARCHIVENAME.txt
# this part | cut -f 5 -d ' ' is sometimes maybe 6 instead of 5, experiment which gives you only the SIZE of the file
echo " " >>stopwatch-$ARCHIVENAME.txt
echo "Started: " $(date -d "1970-01-01 $start sec UTC" +"%A, %d.%m.%Y, %H:%M:%S") >>stopwatch-$ARCHIVENAME.txt #outputs: Sunday, 05.08.2012, 07:16:17
echo "Stopped: " $(date -d "1970-01-01 $stopped sec UTC" +"%A, %d.%m.%Y, %H:%M:%S") >>stopwatch-$ARCHIVENAME.txt
echo "Time needed: " $(($thetime/86400))" days, "$ELAPSEDHRS" hours, "$(($ELAPSSEC/60))" minutes, "$(($ELAPSSEC%60))" seconds ( $thetime secs)" >>stopwatch-$ARCHIVENAME.txt
# outputs: Time needed: 0 days, 0 hrs, 3 minutes, 14 seconds / 194 secs
# outputs: Time needed: 28 days, 2 hrs, 22 minutes, 53 seconds / 2427773 secs


echo "-----------------------" >>stopwatch-$ARCHIVENAME.txt
echo " " >>stopwatch-$ARCHIVENAME.txt
echo "FTP start:" >>stopwatch-$ARCHIVENAME.txt
# again I dont need exact time procedure { time ./lftpscript.sh; } 2>> stopwatch-$ARCHIVENAME.txt
{ ./lftpscript.sh; } 2>> stopwatch-$ARCHIVENAME.txt

ftpstop1=$(date +"%T h ( %s )")
ftpstopped=$(date +%s)

ftptime=$(($ftpstopped-$ftpstarted))
FTPELAPSEDHRS=$((($ftptime%86400/3600)))
FTPELAPSSEC=$(($ftptime%3600))

echo " " >>stopwatch-$ARCHIVENAME.txt
echo "Start of FTP: " $(date -d "1970-01-01 $stopped sec UTC" +"%A, %d.%m.%Y, %H:%M:%S") >>stopwatch-$ARCHIVENAME.txt
echo "End of FTP: " $(date -d "1970-01-01 $ftpstopped sec UTC" +"%A, %d.%m.%Y, %H:%M:%S") >>stopwatch-$ARCHIVENAME.txt
echo "Time of FTP transfer: " $(($ftptime/86400))" days, "$FTPELAPSEDHRS" hours, "$(($FTPELAPSSEC/60))" minutes, "$(($FTPELAPSSEC%60))" seconds ( $ftptime secs)" >>stopwatch-$ARCHIVENAME.txt

mail -s "Backup of $ARCHIVENAME" "email address of recipient" <stopwatch-$ARCHIVENAME.txt
#finally email report :-)

This should do it and you should have a copy of file transferred over FTP to other server.

Otherwise I would use Duplicity for backing up the data, but it's a little more complicated to configure 'include and exclude' ...
Full post / more »»