rsync/ssh without entering password

 

Long time back, I got a task to create DR for one of our production sites. The site  was running on a Ubuntu server and using MySQL and PHP. Replicating DB was not an issue as MySQL was built in with easiest replication technology. But problem was with syncing the Application’s files. We didn’t have any tool to do this task. So I decided to implement it using Linux’s built in rsync command.

But problem was that I have to sync using a cron job and shell script. But rsync asks for password of the user (with which I will connect the primary site), and entering password interactively in my cron job’s shell script was not possible.

The workaround was to set up a pair of ssh keys and put the public key in the remote system's .ssh directory.

For this example I have created one user rsync_user which will connect to the primary server and sync the files.

I will run rsync command on the DR site (Say IP 192.168.10.76), it will connect the Primary Site (Say IP 192.168.10.75) and sync the local directory of the DR site with the remote directory of the Primary site.

 

Steps to be done:

On the computer DR site (Say IP 192.168.10.76)

So on the DR site, login as the user that will connect the primary, in my case I will use my rsync_user user for this purpose.

Now run the command ssh-keygen to generate the keys.

$ ssh-keygen -t rsa

This will create the .ssh directory inside the home directory of the user (if already not present). When this command asks for passphrase, just press enter, do not enter any passphrase.

The above command will create a private key and a public key.

id_rsa is the private key and check the permission for this key. If the permission is not 600 then change the permission.

chmod 600 id_rsa

Now copy the id_rsa.pub file to the Primary Site (this is where I will connect using rsync).

 

On the computer Primary site (Say IP 192.168.10.75)

I have copied the id_rsa.pub file to the home directory of the rsync_user.

If .ssh folder is not present, then create it. Then add the id_rsa.pub key of the DR server into the authorized_keys file. If the file is not present then it will be created by the command cat id_rsa.pub >> .ssh/authorized_keys 

 

That’s it, we are set to connect through rsync/ssh without entering password.

On the DR server, I will run the rsync/ssh command. Before scripting, manually run the command once, so that the primary site is added to the .ssh/known_hosts file.

Now run the rsync command on the DR site.

Also we can connect ssh sessions without entering password.

All set, now I can add the shell script to sync my site’s folder. The sample script is (filename: /home/rsync_user/scripts/files_sync.sh):

echo "File sync started at" >> /home/rsync_user/scripts/files_sync.log
date >> /home/rsync_user/scripts/files_sync.log

rsync -rtvupgo --delete  rsync_user@192.168.10.75:/var/www/ /var/www/ >> /home/rsync_user/scripts/files_sync.log

echo "File sync completed" >> /home/rsync_user/scripts/files_sync.log
echo "" >> /home/rsync_user/scripts/files_sync.log

Create a cron job to do the sync say in every 5 minutes:

Run

crontab –e

and add the following line:

*/5 * * * *  sh /home/rsync_user/scripts/files_sync.sh

We are ready with our requirement

Encrypting MySQL dump

For one of my MySQL databases, I needed to encrypt the MySQL dump files that I used to take for backup of that database. The reason was that the database contained some sensitive information and I did not want to give the MySQL dump files in text format to the backup team.

After some search, I found a nice utility named ccrypt (http://ccrypt.sourceforge.net). This utility can be used to encrypt and decrypt files streams. To install ccrypt you can download it from ccrypt site. As I am using Ubuntu (12.04), so in this example I will use apt-get to install it.

There are many options of ccrypt, you can read the man page of ccrypt for details.

To create our encrypted backup, first I will create a keyfile, which will contain the key phrase(or key phrase). Encryption and decryption depends on this keyword. Longer and stronger keywords provide better security than short and simple ones. By default, the user who runs ccrypt is prompted to enter the keyword. I will use the -k, --keyfile file option of ccrypt, which instructs ccrypt to read the keyword as first line from the specified file.

Create this keyfile as a hidden file and change the permission of the file, so that only the owner of this file can read and write it.

 
Type your key phrase in this file.

Save this file and change its permissions to 600.

Now I will edit the MySQL dump script to enable encryption using ccrypt.

BACKUP_DATE=`date +%k`_`date +%M`_`date +%F`
mysqldump cloud -u pranab -pCl0uD123 | ccrypt -k /root/.backup_keyfile | bzip2 -c > /home/backupadmin/MySQLBackup/bkp_mysql_$BACKUP_DATE.sql.bz2

Now lets examine the backup created using this script.

Decompress the file
#bzip2 -d bkp_mysql_10_35_2013-03-11.sql.bz2
Examine the content
#pico bkp_mysql_10_35_2013-03-11.sql

Its not readable. That what we wanted .

Now encryption part is over, we have to decrypt the backup so that it can be of some use.

# cat bkp_mysql_10_35_2013-03-11.sql | ccrypt -d -k .backup_keyfile > bkp_mysql.sql

The –d option tells ccrypt to decrypt a file or stream.

Lets examine the decrypted file, yes we can now read the content of the decrypted file.

That’s it, simple enough .

Using Partimage

The Partimage is a Linux utility which can be used to save disk partitions into an image file. It only copies the used blocks of a partition. It can also compress the image file with gzip/bzip2 utilities to reduce the image size. It supports many file systems. Following is the list of supported file systems at the time of writing this document (only the stable supports are included in the list):

Name	Description
Ext2fs/Ext3fs	Linux file system
Reiser3	A journalized and powerful file system
FAT16/32	DOS and Old Windows file systems
HPFS	IBM OS/2 File System
JFS	Journalized File System, from IBM, used on Aix
XFS	Journalized File System, used on Irix

It can be downloaded from http://www.partimage.org/Download or it can be installed using apt-get

I will show the use of partimage with an example.

Scenario:
One of my machine’s OS crashed and I am unable to repair it, I need the data in the disks of the machine. So I booted the machine with Ubuntu Linux live CD and mounted an USB disk. I will clone the partitions using partimage and I will store the disk images into the USB disk. I will restore the partition images into a different machine.



Example:

I have mounted the USB disk to /media/disk folder.

Before starting Partimage use fdisk command to find the information of the existing partitions in the system
# fdisk –l

Note the name of the partition to be cloned using partimage.

To start PartImage run the command
$ sudo partimage or
if you have login as root
# partimage

Creating Image

I am going to clone the sda5 partition. Select sda5 partition in the Partition to save/restore list.
Enter the name and location of the image file (I have specified the location in the USB disk) in Image file to create/use.
The name of the image file is specified as image. The file will be created as image.000. I can split the image file into multiple files (this option is in the next screen), in that case the files will be created as image.000, image.001 and so on.
Select Save partition into a new image file in Action to be done
Partimage can save image over the network using the partimage network support, or using Samba / NFS. I am not going to use this feature in this example.


Press F5 to go to the next screen.





Here we can specify the compression level. I am going with the first option i.e. with no compression. If I select None, in that case also only the used blocks will be in the image file. Suppose my partition is 20GB and only 8GB is used, the image file will be around 8GB. Using Gzip and Bzip2 option I can make the file further smaller.
I can use the fsck program to check the partition before creation of the image using the Check partition before saving option.
I can also specify a description of the image, so that in the time of restoration I do not have any confusion in case I have lots of different partition images.
Also I can split the image file into multiple files if my image file is large by specifying the file size. This is especially useful in case I want to write the image files into CD/DVD media.

Press F5 to go to the next screen.


Enter the description of the partition for future reference.

Press OK after entering the description.



Now partimage will check the partition. If it finds any error it will report with a warning message. We can cancel the process here or we can continue with the partition image creation process.

Before starting the partition cloning process partimage will give a summary of the partition.

 Press Enter key to start the partition image creation process.



Partition creation process:
 

 

After partition image is created, partimage will give the statistical information of the process. Press Enter to exit from partimage.

Restoring Image file

Now I will restore an image file in a different machine. Mount the USB disk which contains the image file into the machine. I have mounted the USB disk in /media/usbdisk folder. I will restore the image file into /dev/hda5 partition which is currently mounted in /data folder.

Start partimage with the command
# partimage

Select hda5 partition (in this partition we will restore the image) in Partition to save/restore.
Enter the name and location of the image file (if we have broken the image files into multiple files like image.000, image.001 etc., then enter only the name of the first image file image.000)
Select Restore partition from an image file in Action to be done

 
Press F5 to go to the next screen.



Always remember to unmount the partition in which we will restore the partition. Otherwise partimage will give the error as shown in the screenshot

Unmount the hda5 partition, also format it to clear all the data in this partition.

After unmounting the partition, start the restoration process again. Select partition, Action to be done and image file and go to the next screen. Partimage will show us the description of the image if we have specified it at the creation of the image file (this will help us to identify the image file properly). If the image is not the correct one, we can exit at this point.

Press Enter to hide the Partition description.


Now we can select some options like simulation of the restoration and erasing free blocks with zero values. Also we can specify what to be done if the restoration is successful.

Press F5 to go to the next screen.


 
Partimage will show the summery of the partition image that we are going to restore.

 
Press Enter to continue.



Now partimage will confirm before starting the restoration process. Press Yes to start restoration process.

Restoration process:

 

After restoration process successfully completed, we will get the statistics of the restoration process.

 Press OK to finish the restoration process.

Its done and that's all for 2day.

জয় আই অসম,

প্রণব শর্মা