A simple autotmated backup system.

ssh+rsync+cron is a good combination for backup.

Let us take a simple scenario like this

Main System Backupsystem
hostname server backup
userid myid bup
directory /home/data /home/backup/data

Other constraints/conditions

a.should be done everyday at 10.30am

b.should be automatic without manual intervention.

Let us do it step by step

Stage 1: Establishing passwordless ssh connection

On server

Login as user

Generate key pairs

myid@server$ ssh-keygen -t rsa

Now .ssh directory will have a file named id_rsa.pub. This is the public key of user myid. Copy this file to backup system using scp.

myid@server$ scp .ssh/id_rsa.pub bup@backup:~/.ssh/server.pub

Enter Password:

Now log onto backup system

myid@server$ ssh bup@backup

bup@backup$ cd .ssh

bup@backup$ cat server.pub >>authorized_keys2

ensure that files in .ssh has read/write for owner and no permission for others and group.

Now log out of backup. You will be back at server.

myid@server$

Now try to login again on backup. You will not be prompted for password.

Stage 2: Creating appropriate rsync command

First let us try the rsync at command prompt in the server.

myid@server$ rsync -e ‘ssh -l bup’ -avzp /home/data/ backup:/home/backup/data

Let me explain the command in detail.

-e option allows rsync to use ssh. Rsync can use its own protocol or rsh or ssh. Here we choose ssh and give the options to ssh like -l login name inside a single quote.

-a this option lets rsync to work in archive mode, so files are copied recursively much like tar.

-v verbose gives details about files transferred

-z use compression

-p preserve permission.

While using rsync the distinction between with and without a / at the end is important when specifying source. If last / is omitted it will create a directory and then copy.

Stage 3:Converting rsync into cron job.

Store the above command in a file say mybackup.sh

myid@server$ cat mybackup.sh

rsync -e ‘ssh -l bup’ -avzp /home/data/ backup:/home/backup/data >>/home/myid/errorlog 2>&1

Here I am directing all messages to myid/errorlog file.

Grant executable permission

myid@server$ chmod u+x mybackup.sh

Now add it to cron

myid@server$ crontab -e

30 10 * * * /home/myid/mybackup.sh >>/home/myid/errorlog 2>>/home/myid/errorlog

In the above line 30 stands for minute and 10 stands for hour. Normally cronjob errors are directed to mail, but we can redirect as done above also.

If everything proper you should have a fully automated backup which is very fast also.

Addon features

a.You can schedule backup more than once say every hour.

b.Improve the shell script for multiple backup depending on date and time

c.Start the backup system through wake on lan, take backup and then shutdown.

7 Responses to A simple autotmated backup system.

  1. rajasuperman says:

    I used such a system to backup ~2-3TB of data over several servers on to a couple of master servers, until I hit rsync’s internal limitations.

    Rsync tracks each file in memory during it’s scan and if you have lengthy path names and ~100-200k files, rsync will fail the scan. This is on 32bit architectures, 64bit may fare better.

    After some research, I finalized on rdiff-backup. It’s similar to rsync, but does not have this limitation. rdiff-backup is implemented in python and is available on most platforms.

    Moving to rdiff-backup from rsync is *very* easy and is highly recommended for these advantages:

    1. Functions like rcs/svn and maintains multiple revisions of backup files.

    2. Last revision is stored in full and if you export this folder read-only using nfs/samba, your users can fetch the last backup without sysadmin intervention (file permissions are maintained, so this is secure). A web front end is also available, but I’ve not used it.

    3. Does compression and deduplication natively as it uses rdiff to store past versions.

    4. Stores any number of backup versions on disk efficiently. I used to store ~60 revisions (daily backups) of a 100GB file system using just 150GB of space.

    5. Deleting, searching, restoring from past versions is easy.

  2. ramanchennai says:

    Raja thanks for highlighting rdiff-backup. rdiff is fine for regular server-grade backup with proper revisioning. But involves some extra work. If changes are large, storing revisions require careful monitoring of the space. You cannot just write once and forget forever. For a simple replication rsync serves best.

  3. Padhu says:

    Raman sir,
    Thank you for rsync. This is i want to make it.

  4. Kenneth Gonsalves says:

    will this back up dot files and directories?

    • ramanchennai says:

      Yes, rsync can do them. It has options to transfer device files, links and also preserve permissions, user etc.

      • Kenneth Gonsalves says:

        are you sure? I did rsync with avz and it did not backup my dot directories

      • ramanchennai says:

        I am sure. Here is my terminal output of a test run

        ram@rhlaptop:~$ mkdir test test1
        ram@rhlaptop:~$ mkdir test/.dotdir
        ram@rhlaptop:~$ touch test/nodotfile
        ram@rhlaptop:~$ touch test/.dotdir/nondotfile_in_dotdir
        ram@rhlaptop:~$ touch test/.dotdir/.dotfile_in_dotdir
        ram@rhlaptop:~$ ls -lRa test
        test:
        total 20
        drwxr-xr-x 3 ram ram 4096 2010-08-09 12:30 .
        drwxr-xr-x 146 ram ram 12288 2010-08-09 12:30 ..
        drwxr-xr-x 2 ram ram 4096 2010-08-09 12:31 .dotdir
        -rw-r–r– 1 ram ram 0 2010-08-09 12:30 nodotfile

        test/.dotdir:
        total 8
        drwxr-xr-x 2 ram ram 4096 2010-08-09 12:31 .
        drwxr-xr-x 3 ram ram 4096 2010-08-09 12:30 ..
        -rw-r–r– 1 ram ram 0 2010-08-09 12:31 .dotfile_in_dotdir
        -rw-r–r– 1 ram ram 0 2010-08-09 12:30 nondotfile_in_dotdir
        ram@rhlaptop:~$ rsync -av test/ test1/
        sending incremental file list
        ./
        nodotfile
        .dotdir/
        .dotdir/.dotfile_in_dotdir
        .dotdir/nondotfile_in_dotdir

        sent 269 bytes received 76 bytes 690.00 bytes/sec
        total size is 0 speedup is 0.00
        ram@rhlaptop:~$ ls -lRa test1
        test1:
        total 20
        drwxr-xr-x 3 ram ram 4096 2010-08-09 12:30 .
        drwxr-xr-x 146 ram ram 12288 2010-08-09 12:30 ..
        drwxr-xr-x 2 ram ram 4096 2010-08-09 12:31 .dotdir
        -rw-r–r– 1 ram ram 0 2010-08-09 12:30 nodotfile

        test1/.dotdir:
        total 8
        drwxr-xr-x 2 ram ram 4096 2010-08-09 12:31 .
        drwxr-xr-x 3 ram ram 4096 2010-08-09 12:30 ..
        -rw-r–r– 1 ram ram 0 2010-08-09 12:31 .dotfile_in_dotdir
        -rw-r–r– 1 ram ram 0 2010-08-09 12:30 nondotfile_in_dotdir
        ram@rhlaptop:~$

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: