Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon.
It offers a large number of options that control every aspect of its behavior and permit very flexible specification of the set of files to be copied. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Rsync is widely used for backups and mirroring and as an improved copy command for everyday use.
Rsync finds files that need to be transferred using a "quick check" algorithm (by default) that looks for files that have changed in size or in last-modified time.
Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the file's data does not need to be updated.
I will show shortly how to:
- backup your files from remote to local using rsync
- use modules to have multiple backups possible
- secure the connection with ssh
- avoid to prompt for the password, so that your backup can be done via script/cronjob
Before we start, I'll call "local" the computer where the files have to be copied and "remote" the computer where those files are stored and where you have to listen for ssh connections.
Remote host
To secure our data, we'll use rsync via a remote ssh connection, so there's no need to start rsync as a daemon, but sshd must be configured to accept connections without password and rsa-key authentication must be enabled in your /etc/ssh/sshd_config file:
PermitRootLogin without-password
PubkeyAuthentication yes
AllowUsers root
Here "root" is the only user who is allowed to connect via ssh. So the user "root" will be used at the ssh level and should not be confused with "rsync-user", which will be used to log-in to the rsync "module", site1 in the following example.
Log-in as "root" and create the config file /etc/rsync.conf.
# common stuff
motd file = /etc/rsyncd_motd
# the following in case you want to test rsync as daemon
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsyncd.lock
[site1]
# this is the path of the files to backup
path = /home/ssh-user/path/where/site1/files/live
comment = site1 files
uid = root
gid = root
read only = yes
list = yes
auth users = rsync-user
secrets file = /root/rsyncd.scrt
# we don't have super user access
use chroot = false
[site2]
[....site2 stuff....]
uid and gid are the userID and the groupID under which file transfers will take place.
Before the transfer will start, you have to authenticate rsync with "auth user". Create the secret file ~/rsync.scrt which holds the user:password couples:
rsync-user:password
rsync-user2:password2
Remove the 'r' flag to other users:
chmod o-r ~/rsync.scrt
Local host
Since we want to backup our files by means of a script and a cronjob, it's important that the remote ssh connection will not prompt for any password. We can achieve this by exchanging a ssh-key between client and server.
Create the private and public keys:
root@localhost:~# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa_remoteHost):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa_remoteHost.
Your public key has been saved in /root/.ssh/id_rsa_remoteHost.pub.
The key fingerprint is:
a0:53:33:c5:d1:ea:4c:e2:a1:98:d9:ba:b0:e8:5f:90 root@localhost
The key's randomart image is:
+--[ RSA 2048]----+
| o++o |
| o. . |
| . .. |
| .oo. |
| E.O .S |
| * * |
|. . o . |
|.o. . . |
|+.oo |
+-----------------+
Now you have to append the public key id_rsa_remoteHost.pub to the remote server's ~/.ssh/authorized_keys file. ssh-copy-id is a program which can do this for you:
root@localhost:~# ssh-copy-id -i ~/.ssh/id_rsa_remoteHost -p 12345 root@remoteHost
You will be prompted to enter the root password in order to copy the key.
Now test that the connection is allowed with no password:
root@localhost:~# ssh -p 12345 -l root -i /root/.ssh/id_rsa_remoteHost <remoteHost>
Last login: Mon Sep 2 16:04:57 2013 from localhost
Linux 2.6.32.10-vs2.3.0.36.29.2-smp.
root@remotehost:~#
You can disable the access with password for the user root in your /etc/ssh/sshd_config:
PermitRootLogin prohibit-password
Now we are ready to create our backup shell-script as /usr/local/bin/rsync_backup.sh:
#!/bin/sh
/usr/bin/rsync \
-avz --exclude "*~" --delete-after \
-e "ssh -p 12345 -l root -i /root/.ssh/id_rsa_remoteHost" \
--password-file /root/remoteHost_rsync_pwd \
rsync-user1@::site1 \
/local/destination/path
Remember to give the flag +x to that file:
chmod +x /usr/local/bin/rsync_backup.sh
The password file /root/remoteHost_rsync_pwd holds the password of the rsync connection; in this way our shell-script will not receive a password prompt when it connects. It should be stored in a safe place and priviledges must be given only to the root user. It will contain just the password string.
Maybe the line
-avz --exclude "*~" \
deserves some description, but you are invited to refer to the man page for more details.
- --exclude "*~" is to avoid the copy of backup files of my text editor
- "-a" stands for -rlptgoD and preserves everything
- "-r" means recursive mode while traversing directories
- "-p", "-o", and "-g" preserve the permissions, owner and group information of files and directories to be copied
- "-t" preserves the file and directory timestamps
- "-l" preserves the symbolic links
- "-D" preserves devices and special files
- "-v" turns on verbosity in output
- "-z" enables compression
If you are wondering if the above method of using rsync is suitable for the vpopmail maildirs as well, the answer is yes, but with some adjustments. This is what I have in my backup scripts:
rsync -a \
--stats --delete-after --delete-excluded --numeric-ids --partial \
--exclude=Maildir/tmp/ \
--exclude=Maildir/*/tmp/ \
--exclude=dovecot* \
--exclude=*.lock \
--exclude=*.lock.* \
--exclude=/cache/ \
--exclude=*.qmail \
--exclude=*.qmails \
-e "ssh -i /root/.ssh/id_ed25519 -o StrictHostKeyChecking=no" \
root@${MAIL_IP}:/home/vpopmail/domains/ /home/backup/backup-domains/ \
>> "$LOGFILE" 2>&1
As you can see, I'm excluding the dovecot indexes and all the Maildirs' tmp dirs. This avoids transferring constantly changing temporary or volatile data, reduce the risk of inconsistencies if the backup is restored to a server with a different version of dovecot and improve synchronization performance by avoiding large amounts of non-critical files.
--stats prints statistics of the transfer in the log file.
--delete-after deletes files on the destination server only after the transfer is complete. If the transfer is interrupted midway, you don't immediately lose files on the backup.
--delete-excluded deletes from the backup everything that is no longer included in the sync, even if you had voluntarily excluded it.
--numeric-ids forces rsync to use numeric UIDs and GIDs instead of user and group names. This avoids problems if users on the backup server have different names than on the source server.
--partial Allows partially transferred files to be kept if the transfer is interrupted. With --partial, the file remains, and rsync can resume where it left off next time, saving time and bandwidth.
Connecting to the remote Host
You can have a quick connection to the remote Host if you setup a ~/.ssh/config file as follow
Host MyHost
HostName remoteHost.net
User ssh-user
Port 12345
IdentityFile ~/.ssh/id_rsa_remoteHost
and connecting as
> ssh MyHost
Enter passphrase for key '/home/ssh-user/.ssh/id_rsa_remoteHost':
Last login: Mon Sep 2 16:04:57 2013 from localhost
Linux 2.6.32.10-vs2.3.0.36.29.2-smp.
ssh-user@remotehost:~#
At this point it is convenient to disable root remote access setting /etc/ssh/sshd_config as follow:
PermitRootLogin without-password
AllowUsers ssh-user
PubkeyAuthentication yes
Accedi
