BlogFish

I received some mails about how to exactly configure rsnapshot/rsync to use another user than root to do remote backups as described in my article.

Im my article I wrote:

But using root for this work is not secure because the ssh-key can not be protected by a passphrase as it is used automatically in the background. Therefore create a backup-user on the remote host for this task and allow him to use sudo rsync without a password in /etc/sudoers. Really paranoid can then also allow only this command in .ssh/autorized_keys. See the rsnapshot mailling list for more information on how to do this.

The danger of using root for the backups is that when somebody hacks your backup-server he will gain access to all SSH private keys. With these he will be able to login to your servers as the private keys are configured to use no password. And if the user that for that key on the remode side is root, you’re lost. As the information on the rsnapshot mailing list is maybe not enough, I will describe in detail how to configure rsnapshot without using root. A hacker with access to the SSH private keys will only be able to backup your files.

First create a backup user on the production server (that means the server you want to backup).

remote# adduser

Enter the desired information. Make sure that this user in not in the group wheel as this should be a non-priviledged user, that is the hole reason behind this article.

As described in this article on securityfocus.com, create a ssh-key for the user on the local machine and copy the public key to the remote server.

local# ssh-keygen -t rsa
local# scp id_rsa.pub ssh-remote-server:id_rsa_backup.pub
local# ssh ssh-remote-server
remote# cat id_rsa_backup.pub >> authorized_keys

The last command will tell sshd to allow connections without a password to this account if they have the private key to this public key. So far, nothing new. Now instead of using root we ca now login with the user backup without a password to the remote host. So lets tell rsnapshot to use our new backup user:

Change

backup root@www.example.com:/etc/ www.example.com/etc/

to

backup backup@www.example.com:/etc/ www.example.com/etc/

Are we done now? Unfortunately no as the user backup has not the rights to do the backup. He is not root and therefore does not have access to many files like special files in /etc or the files of other users.

We need to allow the backup user to have access to all files when using rsync .
Therefore we need to install sudo on the remote server and allow the backup user to call sudo rsync without using a password.

On FreeBSD

remote# cd /usr/ports/security/sudo
remote# make install clean
remote# echo “backup remote = NOPASSWD: /usr/local/bin/rsync_real” >> /usr/local/etc/sudoers

Of course you should use visudo to edit sudoers (the config file of sudo).
This line will allow the local user backup to use the command /usr/local/bin/rsync_real as root without a password.

But wait, why rsync_real and not rsync? rsnapshot will call rsync but we have to call sudo rsync in order to gain root privileges for this command. So we have to create a wrapper script that will call sudo rsync for rsnapshot as rsnapshot is not aware of sudo.

remote# cd /usr/local/bin/
remote# mv rsync rsync_real

Now create the wrapper script:

remote# cat /usr/local/bin/rsync
#!/bin/sh
/usr/local/bin/sudo /usr/local/bin/rsync_real ”$@”

Now rsnapshot should be able to use the remote user backup to backup using rsync over ssh. But we can do more. As I wrote:

Really paranoid can then also allow only this command in .ssh/autorizedkeys

Edit /home/backup/.ssh/authorized_keys to allow the backup user that is logged in through the ssh private key only a certain command and to connect form only certain hosts.

remote# cat .ssh/authorized_keys
from=”backup.example.com”, command=”/home/backup/validate-rsync.sh” ssh-rsa AADDB39z….....

Why not allowing /usr/local/bin/rsync? Because sshd will only allow the command as written in the authorized_keys file, the problem is that we want to call rsync with different arguments and this is not possible with the command=”” directive in authorized_keys. Therefore we have to create another wrapper and call the real (better the other wrapper) rsync. Luckily sshd will set the original command issued over SSH as an variable () that we can use in the vaildate-rsync.sh script.

I got the following validate-rsync.sh from here (notice my change in the rsync-case).

remote# chown backup validate-rsync.sh
remote# chmod u+x validate-rsync.sh
remote# cat validate-rsync.sh
#!/bin/sh
case ”$SSH_ORIGINAL_COMMAND” in
&)
echo “Rejected”
;;
;*)
echo “Rejected”
;;
rsync)
$SSH_ORIGINAL_COMMAND
;;
*)
echo “Rejected”
;;
esac

Puh! Hard work for the paranoid. Let’s walk through the hole process again.

The backup machine will connect to the remote server through rsync over ssh using the user backup (backup@remote in rsnapshot.conf). The connection will use the private key of the user on the backup machine to identify itself to the remote sshd (.ssh/authorized_keys). The SSH server will only allow this connection form certain hosts (from-directive) and limit the user to the validate-rsync.sh command (command-directive).

Now the validate-rsync.sh script will call rsync with the arguments that the backup user originally used (rsnapshots commands). But we have to use sudo in order to gain access to all files so rsync is another wrapper around sudo rsync_real.

In the end rsync_real (the real rsync :-) is called with super-user rights and will actually do the backup.

More complicated then just using root? Yes
More secure? Definitely!

Truly paranoid can further edit the validate-rsync.sh script to validate the exact rsync calls (check rsnapshot.conf). Now a hacker will only be able to backup your files to your server!

UPDATE:
There is an interesting thread on the rsnapshot mailing list with some improvements on my HOWTO.
Thanks to Jason for pointing it out.

If you ever wanted to have scheduled remote backups but Amanda or Bacula were too complicated for your needs and just cron jobs with rsync were not enough, you should have a look at rsnapshot.

I started with using rsync over ssh to backup my servers and workstations to a central backup server.

rsync -vaz—del user@remote:/etc /backups/remote/etc

The problem with his approach is that with rsync it is easy to have a synchronized version of your live system but when you want to be able to access several version to the past it is getting complicated. You have to write shell scripts and cron jobs in order to have several versions of the live systems like backing up every system once a week and keeping up to 4 weekly backups. Then maybe you want to also keep 12 monthly backups and so on. Your scripts are getting bigger and bigger and disk space is starting to shrink.

rsnapshots is trying to solve these problems. It is a remote filesystem snapshot utility based on rsync. It can do all the things I described while keeping the needed disk space at a minimum through using hard links. Therefore if in your weekly backups of /etc only 3 file changed, only these files will be copied but the rest will also be accessible as hard links to the backup store where they last changed.

That means that I can have daily/weekly/monthly/yearly snapshots of whole filesystems but the used disk space will be minimal.

Configuration is very easy. All configuration is done on the backup server. This server will use rsync over ssh to connect to the servers that should be backed up and copy the given filesystem(-changes) to the backup store.

After installing rsnapshot (and rsync), edit rsnapshot.conf. First set where you want to store the backups and enable cross-plattform backup of special files (e.g. FIFOs):

snapshot_root /backup/snapshots/
link_dest 1

We are not done with the general configuration. Now we have to specify the intervals for backing up. The default of hourly/daily/weekly/monthly is reasonable and useful and will not hurt your disk space because of the hard links. For my demands, hourly is not needed, so I comment it out:

#interval hourly 6
interval daily 7
interval weekly 4
interval monthly 3

When you use your own intervals be sure to use the smallest first. Larger intervals use the smaller ones as a start. That means that after 4 weeks the oldest weekly snapshot will become the newest monthly backup. The numbers mean that the hourly snapshot will be taken 6 times a day (or every four hours), the daily snapshot will be taken 7 times a week and so on.

Now just tell rsnapshot what to backup and where to put it:

# localhost
backup /root/ localhost/root/
backup /etc/ localhost/etc/
backup /usr/local/etc/ localhost/usr_local_etc/

This will backup the directories /root, /etc, and /usr/local/etc to a directory named localhost in snapshot_root on the backup machine. For remote backups just give ssh the user and host:

# www.example.com
backup root@www.example.com:/root/ www.example.com/root/
backup root@www.example.com:/etc/ www.example.com/etc/
backup root@www.example.com:/usr/local/etc/ www.example.com/usr_local_etc/
backup root@www.example.com:/usr/local/www/ www.example.com/usr_local_www/
backup root@www.example.com:/usr/local/home/ www.example.com/home/

The use the hostname of the remote host as the directory name on the local machine is a often used convention.

For this to work, the user root must be able to log in to www.example.com with a RSA or DSA authentication key. Use ssh-keygen to create these or read here for more information on how to do this. Further root has to be able to read all the directories on the remote server. This is why we start with root.

But using root for this work is not secure because the ssh-key can not be protected by a passphrase as it is used automatically in the background. Therefore create a backup-user on the remote host for this task and allow him to use sudo rsync without a password in /etc/sudoers. Really paranoid can then also allow only this command in .ssh/autorizedkeys. See the rsnapshot mailling list for more information on how to do this.

rsnapshot can also run scripts and backup their output. This is useful for backup of databases, e.g. the script runs pg_dump and rsnapshot backups the dump. Sadly this can only be done for local systems, on remote systems you need a cron job that dumps the database and then rsnapshot can backup the directory with the dump.

We are now nearly finished, the only thing left is to add the cron jobs that call rsnapshot. The daily cron job has to run once a day and just call

rsnapshot daily

The weekly cron job has to run one a week and call

rsnapshot weekly

You get the picture, just call rsnapshot and tell it which interval to back up. On FreeBSD there are already the directories /etc/periodic/{daily, weekly, monthly}. Just insert your script there and it will just work.

After running rsnapshot for some weeks, your backup directory should look like this:

$ cd /backup/snapshots/
$ ls -l
... daily.0
... daily.1
... daily.2
... daily.3
... daily.4
... daily.5
... daily.6
... weekly.0
... weekly.1
... weekly.2
... weekly.3
... monthly.0
$ ls -l daily.0
... localhost
... www.example.com
$ ls-l daily.0/localhost
... etc
... root
... usr_local_etc

I can so access snapshots of whole filesystems of all my servers and workstations several weeks and months to the past without wasting disk space. And the configuration took me only 15 minutes!

See the official HOWTO for a more detailed guide and information on giving normal users access to the snapshots through Samba of NFS.

UPDATE:

See my follow-up article for more information on how to use another user than root for the backups.

Yesterday I bough a new 200 GB hard disk for my FreeBSD server. I wanted to use it for backups and private data so I thought of using an encrypted filesystem.

The Geom Based Disk Encryption GBDE is the perfect candidate for such things on FreeBSD. It makes encrypting filesystems and using them transparently very easy.

At first I connected the disk to my server and created a partition on it. I was lazy so I used sysinstall. Normally one would use fdisk to create a slice and bsdlabel to create a partition. This is what sysinstall is doing behind the scenes. Refer to the handbook for how to attach a new drive to FreeBSD.

For the next steps I used the man page of GBDE and the handbook as a reference. Beware the handbook is a litte out-dated for the new auto-mount features but for me it did the job.

After sysinstall I had a new partition on ad1s1 (second IDE disk, fist slice):

bsdlabel ad1s1
# /dev/ad1s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
c: 390716802 0 unused 0 0 # “raw” part, don’t edit
d: 390716802 0 4.2BSD 2048 16384 28552

So ad1s1d is my “raw” partition for GBDE. You have to init the partition with GBDE. If you used sysinstall, be sure that /dev/ad1s1 is not mounted because sysintall will mount the labeled disk to the specified mount point.

gbde init /dev/ad1s1d -L /etc/ad1s1d.lock

/etc/ad1s1d.lock is the lockfile for GBDE. Be sure not to publish it. This is not the keyfile, only lock sector data but its presence would make the attackers job easyer. DO NOT EVER DELETE THIS FILE OR YOUR DATA IS LOST. Now you have to enter the password twice and be sure to remember it or again: all your data on this disk will be lost, when you forget it.

After initializing you have to actually attach the drive in order to make it usable for the system.

gbde attach ad1s1fd-l /etc/ad1s1d.lock

This step will create the device /dev/ad1s1d.bde, the accessible filesystem. Now I have a “normal” partition on that I can create a filesystem on:

newfs -U -m 3 /dev/ad1s1d.bde

Notice the -m 3, this will reserve only 3% of the filesystem for the minimum free space threshold (default is 8%). At first I used the default only to notice that I lost 15GB to this “reserve”. Quite a big reserve I though and lowered it to 5GB. The -U will enable Softupdates on the filesystem.

Now the filesystem is ready and can be mounted.

mkdir /encrypted
mount /dev/ad1s1d.bde /encrypted

In order to detach it, use the following command:

umount /encrypted
gbde detach ad1s1d

This was it, from now on you can use the encrypted filesystem with gbde attach, mount, ..., umount, gbde detach. I also included the umount and gbde detach calls to my /etc/rc.shutdown just to be sure.

If you want to mount the encrypted volume on boot modify /etc/fstab

/dev/adds1d.bde /encrypted ufs rw 0 0

and /etc/rc.conf:

gbde_devices=”AUTO”

You will be prompted on boot for the password. If you do not use this procedure, fsck will not check the encrypted disk, so be sure to include a fsck /dev/ad1s1d.bde before mounting the filesystem.

While I was doing this and reading through GBDE I came across a nice way to easily encrypt swap. Just add a ”.bde” to your swap entry in /etc/fstab:

/dev/ad0s1b.bde none swap sw 0 0

And add this line in /etc/rc.conf

gbde_swap_enable=”YES”

That was easy, wasn’t it?

For more nice features of GBDE like using several keys for decryption refer to the man page. Further I found this german Wiki entry usefull.