Feb 6, 2020

Incremental Backup with TAR / simple FTP to another location and email status

This is my script for archiving incremental and full backup with TAR in Linux and then FTP that archive to another 'server' for security if server loses data.
This is not complete backup to make it possible to restore 'bare metal' for that case use Mondo Backup.

The script requires GNU TAR which is capable of Incremental archiving

The LFTP improved FTP client for Linux capable of auto retrying the transfer until finished successfully

After I spent some time discovering The BIG BANG of Universe and The Meaning of Life :lol I managed somehow to create a script to make some backup of files on server and TAR/GZIP it and then FTP the archive to another FTP server and finally email the results.
This script also measures time needed to complete it and deletes archive older than xx days (set in find -mtime +20) and makes incremental backup every weekday and then FULL BACKUP on Sundays (which suits me bcoz no heavy load).

This is the script I had written to work for ME (you will have to modify it for yourself, I hope you find what and where), since I put it in CRON making it run every day

Put the scripts (files) to some directory where you will be making backups to, I use


Files for TAR to include and exclude are in plain txt format and filenames listed each name in separate line (these paths will be included in TAR-GZIP archive):
file: including.txt:

For excluding the files / directories the syntax is:
- var/tmp  <-- exclude directory matching 'var/tmp' in the name
- spool  <-- exclude files matching 'spool' in the name: e.g. 'spool1 spoolwhatever' also *_log* matches names including '_log'
- var/tmp/serverbackups  <-- exclude directory with backups in it so we don't archive ourselves when creating new archives - obviously !

file: excluding.txt:

I'm using LFTP to make sure that FTP transaction runs complete since ftp didn't always finish, the script for transferring the BACKUP file:
FILE=$(cat archivename.txt)

lftp -c "open $HOST && user $USER $PASSWD && cd FOLDER_NAME_FOR_STORING/backups/ && put $FILE" <<END_SCRIPT
# all in one command that connects to HOST=ftp.domain.com with Username/password
# and changes directory to whatever you need
# then transfers the file with the name of created archive in archivename.txt (ex. BACKUP-2009-12-15-Fri-01-15-01h.tgz)
exit 0

Crontab running the script, this is located in PATH '/usr/local/bin':

0 1 * * * archive.sh >/dev/null         # runs the script at 1.00AM every day
runs the script archive.sh which only changes the directory path to '/usr/tmp/serverbackups' where the real Backup script is located.
Then it runs the backup script.
file: archive.sh:
cd /usr/tmp/server_backup
./backup.sh         # the real script to make backup

Now comes the 'real' script which handles the archiving with TAR and sends over FTP to another location.

file: backup.sh:

# DELETE archive older than -mtime +'days'
find . -name 'BACKUP*.tgz' -mtime +20 -delete
find . -name 'stopwatch*' -mtime +2 -delete

start1=$(date +"%T h ( %s )")
start=$(date +%s)

# on SUNDAY make FULL backup
if [ $(date +"%a") = "Sun" ]; then
SNAPSHOTFILE="./usr-full"; # needed by TAR (GNU-TAR to be precise) which is used to compare for incremental backups
ARCHIVENAME=BACKUP-full-$(date +"%F-%a-%H-%M-%Sh").tgz; # self explaining
rm -f "./usr-full";

ARCHIVENAME=BACKUP-$(date +"%F-%a-%H-%M-%Sh").tgz;
# a name of a backup file: BACKUP-2009-12-15-Fri-01-15-01h.tgz
cp "./usr-full" "./usr-1";

echo $ARCHIVENAME >archivename.txt # need the name to FTP transfer so store it in file archivename.txt which doesn't change (used later in lftpscript.sh ) !
# creating text to send in email
echo "-----------------------" >stopwatch-$archivename.txt
echo "Backup of $ARCHIVENAME" >>stopwatch-$archivename.txt
echo "-----------------------" >>stopwatch-$archivename.txt
echo " " >>stopwatch-$archivename.txt # echo " " makes new line /CR or LF whatever it does
# I do not need this precise time { time tar -T including.txt -X excluding.txt -pczvRf $ARCHIVENAME; } 2>> stopwatch-$ARCHIVENAME.txt >/dev/null
{ tar -T including.txt -X excluding.txt -pczvR --listed-incremental=$SNAPSHOTFILE -f $ARCHIVENAME; } 2>> stopwatch-$ARCHIVENAME.txt >/dev/null

stopped1=$(date +"%T h ( %s )")
stopped=$(date +%s)

thetime=$(($stopped-$start)) # doing some math in shell that's why $()

echo " " >>stopwatch-$ARCHIVENAME.txt
echo -n "File Size (Byte-s) : " >>stopwatch-$ARCHIVENAME.txt
ls -al "$ARCHIVENAME" | cut -f 5 -d ' ' >>stopwatch-$ARCHIVENAME.txt
# this part | cut -f 5 -d ' ' is sometimes maybe 6 instead of 5, experiment which gives you only the SIZE of the file
echo " " >>stopwatch-$ARCHIVENAME.txt
echo "Started: " $(date -d "1970-01-01 $start sec UTC" +"%A, %d.%m.%Y, %H:%M:%S") >>stopwatch-$ARCHIVENAME.txt #outputs: Sunday, 05.08.2012, 07:16:17
echo "Stopped: " $(date -d "1970-01-01 $stopped sec UTC" +"%A, %d.%m.%Y, %H:%M:%S") >>stopwatch-$ARCHIVENAME.txt
echo "Time needed: " $(($thetime/86400))" days, "$ELAPSEDHRS" hours, "$(($ELAPSSEC/60))" minutes, "$(($ELAPSSEC%60))" seconds ( $thetime secs)" >>stopwatch-$ARCHIVENAME.txt
# outputs: Time needed: 0 days, 0 hrs, 3 minutes, 14 seconds / 194 secs
# outputs: Time needed: 28 days, 2 hrs, 22 minutes, 53 seconds / 2427773 secs

echo "-----------------------" >>stopwatch-$ARCHIVENAME.txt
echo " " >>stopwatch-$ARCHIVENAME.txt
echo "FTP start:" >>stopwatch-$ARCHIVENAME.txt
# again I dont need exact time procedure { time ./lftpscript.sh; } 2>> stopwatch-$ARCHIVENAME.txt
{ ./lftpscript.sh; } 2>> stopwatch-$ARCHIVENAME.txt

ftpstop1=$(date +"%T h ( %s )")
ftpstopped=$(date +%s)


echo " " >>stopwatch-$ARCHIVENAME.txt
echo "Start of FTP: " $(date -d "1970-01-01 $stopped sec UTC" +"%A, %d.%m.%Y, %H:%M:%S") >>stopwatch-$ARCHIVENAME.txt
echo "End of FTP: " $(date -d "1970-01-01 $ftpstopped sec UTC" +"%A, %d.%m.%Y, %H:%M:%S") >>stopwatch-$ARCHIVENAME.txt
echo "Time of FTP transfer: " $(($ftptime/86400))" days, "$FTPELAPSEDHRS" hours, "$(($FTPELAPSSEC/60))" minutes, "$(($FTPELAPSSEC%60))" seconds ( $ftptime secs)" >>stopwatch-$ARCHIVENAME.txt

mail -s "Backup of $ARCHIVENAME" "email address of recipient" <stopwatch-$ARCHIVENAME.txt
#finally email report :-)

This should do it and you should have a copy of file transferred over FTP to other server.

Otherwise I would use Duplicity for backing up the data, but it's a little more complicated to configure 'include and exclude' ...


^F.B said...

Works well with me! but i have this error when the backup.sh run,

[root@j0 test]# sh /usr/bin/archive.sh
cp: cannot stat `./usr-full': No such file or directory
block 0: /backup/test/worksfilesbackup/
[root@j0 test]#


Han Solo said...

I think that happens every time until one (1) full backup is done. Then it will be ok.

^F.B said...

I think that two, thanks very much.

But if i have a Hetzner backup and the node have VM's ( Virtualizor Panel ), should i backup full system / so i won't exclude anything or should i exclude some files ?


Han Solo said...

No, don't make a / backup, "full backup" term is meant the whole directory you want to backup (can be /home/username/documents) with all the files and directories in it.

and for backup it's like this:

SNAPSHOTFILE="./usr-full"; # needed by TAR (GNU-TAR to be precise) which is used to compare for incremental backups
ARCHIVENAME=BACKUP-full-$(date +"%F-%a-%H-%M-%Sh").tgz; # self explaining

and since today it's not Sunday the full backup of your directory was not made.
You should do a manual full backup of your directory with:

tar -T including.txt -X excluding.txt -pczvR --listed-incremental=./usr-full -f BACKUP-full-$(date +"%F-%a-%H-%M-%Sh").tgz

then it will create a usr-full file which will be used as a reference what files has changed after this backup taken.

^F.B said...

Thanks Han,

Please give me your advise regard my backup plan.

I have a node were i have Virtulizor CP installed and have more than 5 VPs i would like to backup them to the HD, but the node don't have any webserver i would to backup only the VPS i have run inside it.

so am lost here what to backup in the includes.txt and what to exclude.txt them, i have another servers out of this server were i make remotely backup to them to /remote-backup folder so may i use only in the include.txt the /remotely-backup and the VPSs i would backup them to the HD.

So should i put in includes.txt


Thanks again for your helpful thread but i would like to have your expert suggestion regard my backup plan here.


Han Solo said...

I'm sorry,
I'm not familiar with Virtualizor
where and what files does it have.

I would try to log into a virtual server through SSH like Virtualizor demo has and then do
# ls -al /

or whatever directory you may want to list
to find out where are the files stored you want to backup.

If you want to backup the Virtual machines, then you will have to login to your Host machine (which hosts all VM's) and try to find out where are the files that VM's run from. Then include that into backup.

good luck.

^F.B said...

It's okay, but if i want only to backup


and exclude everything so should i only put in the exclude.txt


and include.txt


am i right?

Han Solo said...

No, don't
should then be empty, because you don't want to exclude any files/dirs under the "include" path.

or just don't use "-X excluding.txt" within the TAR line:
like this:
tar -T including.txt -pczvR --listed-incremental=$SNAPSHOTFILE -f $ARCHIVENAME;

This way you will backup EVERYTHING within:

and nothing else (like you mean: / or /etc ).

Exclude works UNDER the include path's:

include: /remotely-backup

which contains:

and if you want to exclude test2, then you would write it into exclude.txt:

this will backup:

^F.B said...

I'll test and let you know, thanks a bunch brother!

^F.B said...

Thanks Han, everything seems to working fine for backup and send it to the ftp server, but i have big issue here were i cannot log in my 100GB ftp server outside Hetzner Network, they only accept connection inside them network, so i did make authentication between my box and the HD backup server via this link;


but the problem now i don't have any idea how to restore files from HD to my server so i can restore it?

Han Solo said...

since you can connect to your HD (via FTP) only from your server then try with FTP to get the backed up file (like: BACKUP-full-2011-11...tgz) and retreive it on your server,
then unpack it with TAR into some test directory (i.e. /var/tmp/restored_backup:
# tar xzvf BACKUP-full-....tgz

see what you get in that directory must be all the files:

if you set to include these directories in "include.txt"

^F.B said...

Dear Han,

I did move files from backup HD to the srv using the command;

wget ftp://uxxxxx:pass@uxxxx.your-backup.de/stuff/BACKUP-2011-11-22-Tue-14-38-53h.tgz

but i did try to extract it using your command but that doesn't work!?

[root@j0 ]# tar xzvf BACKUP-2011-11-22-Tue-14-38-53h.tgz
[root@j0 ]# ls -la
total 12
drwxr-xr-x 2 root root 4096 Nov 22 15:14 .
drwxr-xr-x 3 root root 4096 Nov 22 15:13 ..
-rw-r--r-- 1 root root 45 Nov 22 15:14 BACKUP-2011-11-22-Tue-14-38-53h.tgz
[root@j0 ]#

and tar -xvvf *.tgz the same it's doesn't show any files or show that the file has been extracted?

Han Solo said...

your file length indicates that it's empty

drwxr-xr-x 3 root root 4096 Nov 22 15:13 ..
-rw-r--r-- 1 root root 45 Nov 22 15:14 BACKUP-2011-11-22-Tue-14-38-53h.tgz

this 45 is file length, which is just as the filename and nothing in it.

You can view what is in the archive with:
# tar tzvf BACKUP-2011-...

but I think that you have downloaded an incremental backup which is empty because no files changed or something.
Try download BACKUP-FULL... file

^F.B said...

i did read the stopwatch and some error there;

[root@j0 backup]# cat stopwatch-BACKUP-2011-11-22-Tue-16-00-32h.tgz.txt
tar: : Cannot stat: No such file or directory
tar: Substituting `.' for empty member name
tar: : Cannot stat: No such file or directory
tar: Error exit delayed from previous errors

File Size (Byte-s) : 45

Started: 16:00:32 h ( 1321970432 )
Stopped: 16:00:32 h ( 1321970432 )
Time needed: 00:00:00 / 0 secs

FTP start:

Start of FTP: 16:00:32 h ( 1321970432 )
End of FTP: 16:00:32 h ( 1321970432 )
Time of FTP transfer: 00:00:00 / 0 secs
[root@j0 backup]#

Han Solo said...

Try running TAR manually:

@/usr/tmp/server_backup/# tar pczvRf BACKUP-test-2011-11-22.tgz /remotely-backup /vz/

then you must get the archive file BACKUP-test.... filled with files located in /remotely-backup and /vz

if so, proceed manually running with -T include.txt file to see if you get same results (files archived from /remotely-backup and /vz)

and so on...

^F.B said...

Dear Han,

Now i had test everything and it seems working fine with me, even the tar compress i just note that i was mistaking to include
files were i need to backup in exclude.txt by mistake were i should put it in include.txt and leave exclude.txt empty.

I had check everything and it seems well but the problem that the backup is take a space from my server were i need to know
if we make the crontab work with this script it well delete the backup every 20 days, but when i make a backup of all
vps i have (8 VPSs) in /vz/private/ folder it take more than 2 GB, so i need to make this script to delete the folder from our server
when the transfer is completed and make sure there is no compressed backup in our server after moving the backup to ftp server.

and here my configuration for archive.sh were i put it in /usr/bin/ binary and then chmod +x archive.sh and make it as
you mention in this article... is this right now?

cat /usr/bin/archive.sh;nano /etc/crontab
cd /backup/backup
./backup.sh # the real script to make backup

# run-parts
01 * * * * root run-parts /etc/cron.hourly
02 4 * * * root run-parts /etc/cron.daily
22 4 * * 0 root run-parts /etc/cron.weekly
42 4 1 * * root run-parts /etc/cron.monthly
0 1 * * * archive.sh >/dev/null #runs the script at 1.00AM every day

should i put it like this or put the full path for the archive. (/usr/bin/archive.sh) in the crontab?


Han Solo said...


I would suggest you to use
/usr/bin/archive.sh >/dev/null
in the crontab

and to change the lines in backup.sh:
# DELETE archive older than -mtime +'days'
find . -name 'BACKUP*.tgz' -mtime +20 -delete
find . -name 'stopwatch*' -mtime +2 -delete


find . -name 'BACKUP*.tgz' -mtime +0 -delete
find . -name 'stopwatch*' -mtime +0 -delete
# (0 = zero, not letter o)
and this way you will leave only current BACKUP file on your server, every other older will be deleted on next run Archive.sh

^F.B said...

Thanks, everything seems to be ok.

^F.B said...

Dear Han,

I have this shell-script

# Local dir location
# remote ssh server
# user
# server IP / host
#remote dir to backup
rsync --exclude '*access.log*' --exclude '*error.log*' -avz -e 'ssh -p 9999' ${SSHUER}@${SSHSERVER}:${SSHBACKUPROOT} ${LOCALBAKDIR}
# log if backup failed or not to /var/log/messages file
[ $? -eq 0 ] && logger 'RSYNC BACKUP : Done' || logger 'RSYNC BACKUP : FAILED!'

This script i make for sending data from remote-server to local-server after making authentication to transfer without password using port 9999.

but i try to compress this files from remove server and send them in file-name +date+time.tgz but i don't know how (newbie in shell-script).

and if you could to add another file to add ( /var/log/messages + /var/log/httpd/access_log ) so the script will transfer them in separate compress files so it would be

file-name +date+time.tgz

That would really helpful to me, and please email me your email # gits(.)systems_at_gmail.com :).

^F.B said...

Dear Han,

I have a question please, now the backup of all server is taking a huge memory, as we do it for 1 time a full backup every Sunday and to save 1 backup only as below:

# DELETE archive older than -mtime +'days'
find . -name 'BACKUP*.tgz' -mtime +1 -delete
find . -name 'stopwatch*' -mtime +1 -delete

but i need to remove the backup automatic when the backup file is transferred to the backup server, how could we do that please?

Han Solo said...


in the backup.sh script at the very end (after:
mail -s "Backup of $ARCHIVENAME" "email address of recipient" < stopwatch-$ARCHIVENAME.txt
#finally email report )

add this to remove:

so the file will be removed each time when backup occurs and will not leave any 'leftovers' after FTP completes transfer and email is sent.

^F.B said...

Thanks Han,

I did that, and i did move the backup.sh in /etc/cron.daily so the backup will run daily, and the full backup will run on Sunday, am i right?

Han Solo said...


Betti es Csaba Finnorszagban said...

Could you please help me modify the script so it would delete the local copies after sending over ftp?

I would like this to store backups on the remote site only.

Han Solo said...

The same as above:

in the backup.sh script at the very end (after:
mail -s "Backup of $ARCHIVENAME" "email address of recipient" < stopwatch-$ARCHIVENAME.txt
#finally email report )

add this to remove the file:

so the file will be removed each time when backup occurs and will not leave any 'leftovers' after FTP completes transfer and email is sent.

Csaba said...

Oh, sorry. I did not realize you already answered this question before.

Would this program delete old backups on the remote location (ftp site) when it deletes old ones locally? That would be cool.

Han Solo said...

Would this program delete old backups on the remote location (ftp site) when it deletes old ones locally? That would be cool.

Sorry, no, that would need another script in FTP transfer to check if "older" exist and then delete them through FTP.

What I did on the FTP server (where backups are stored) I added cron to check for files in /usr/backup-dir
if there is older than specified time, then delete them.

10 2 * * * /usr/backup-dir/delete-backups.sh

cd /usr/backup-dir/
find . \( -name "BACKUP-filename*.tgz" \) -mtime +31 -delete

this deletes "BACKUP-filename*.tgz" older than 31 days (1 month) on the FTP server side.

Csaba said...

Hi Han,

Thanks for helyping and sorry for bothering you again.

cd /usr/backup-dir/
find . \( -name "BACKUP-filename*.tgz" \) -mtime +31 -delete

I tried to run it manually but no luck. What I do not undesrtand is, how would this find the backups? Shouldn't this be instead "BACKUP*.tgz" ? How would this BACKUP-filename*.tgz resolve any file?

Also I have trouble with mtime. Let's say I want backups older than 10 days to be deleted, should I write -mtime +10 ?? Or if older then 1 day -mtime +1 ??

I tried to find good info explaining -mtime but did not find one.

Han Solo said...


Shouldn't this be instead "BACKUP*.tgz" ?
Yes, to both of your Q.

It was my typo with 'filename'

and for mtime examples this document tries to explain it all.

Csaba said...

HI Han,

here is an alternate copy method for scp.
I made this script and named it scpscript.sh
And so if someone needs ssh here it is, just need to change the backup script also.

FILE=$(cat archivename.txt)

if scp $FILE $USER@$HOST:/var/backup/$FILE
echo "ok"
echo "nem ok"


Csaba said...

Hi Han,

sorry to ask again, but..
the backup progrgam always makes a full backup and not any incremental. I am deleting the local copy after ftp transfer.

Do you have any idea how to fix this?


Han Solo said...


well, the first thing that occurs on my mind is that it can't get properly weekday names (Mon, Tue, Wed ...) as it can be your system maybe gives a different Day name.
You can try it if you type in your command line:
# echo $(date +"%a")
# Tue

If it doesn't return that, then you will need to modify this line:
# on SUNDAY make FULL backup
if [ $(date +"%a") = "Sun" ]; then

according to your name of Sunday

Csaba said...

Thanks that did it. I am trying to perfect this script. When I run the script manually even more than once during the day, because it should be using incrementel backup shouldn't this create backups with only the files that are new?
In other words, when I look at backups even a few days old same files are in there in more than one backup.

Is that make sense what I am trying to write down here?


Han Solo said...


I know what you mean, it's called Differential backup which unfortunately TAR doesn't know (automatically).
If you want that, you can take a look at backup solutions, mostly based on rsync.
Some kind of example is written here

Hope it helps...

Csaba said...

Thanks again for helping. For my needs it does not work or I am not good enough a programmer.
So I thought I go back using sbackup and on the remote machine run a cron script to extract tgz files to tars.
May I ask for another help?
I tried this:

cd /usr/backup-dir/
find . \( -name "BACKUP-filename*.tgz" \) | gunzip -delete

but this is obviously not working.

Can you please help me with this?

Han Solo said...


the line
find . \( -name "BACKUP-filename*.tgz" \) | gunzip -delete
is not working because of extra \(

Remove it and write the line like this:
$(find . -name "BACKUP-filename*.tgz" | gunzip -delete )

Csaba said...

Hi, and thanks.
I still need to learn a lot.
I went back to using my original sbackup that better suits my needs and created a little script on the target machine to convert every night the tgz files to tar.
I would like to add a line in the end of the script to delete everything in that folder (and sub folders)that is older than 7 days but do not delete my program :) that is also in that directory.

How would I do that?

I also tried this without putting it to a text file but that did not work. so that I would not have to put it to a text file first.
Can you give me a hint?

BTW. here is my script:


FILE_LIST=$(find "$BACKUP_DIR" -name "*.tgz")

echo $FILE_LIST > lista.txt

filecontent=( `cat "lista.txt" `)

for t in "${filecontent[@]}"
echo $t
gunzip $t

echo "done extracting"
echo "all tgz files deleted"

#delete files and directories that are older than 7 days
#but leave alone this script

# End of script

Han Solo said...


Here's how you can exclude a filename from search:
find . \( -name "script_name" \) -prune -o -mtime +7 -print

from this example