rsync/ssh without entering password

 

Long time back, I got a task to create DR for one of our production sites. The site  was running on a Ubuntu server and using MySQL and PHP. Replicating DB was not an issue as MySQL was built in with easiest replication technology. But problem was with syncing the Application’s files. We didn’t have any tool to do this task. So I decided to implement it using Linux’s built in rsync command.

But problem was that I have to sync using a cron job and shell script. But rsync asks for password of the user (with which I will connect the primary site), and entering password interactively in my cron job’s shell script was not possible.

The workaround was to set up a pair of ssh keys and put the public key in the remote system's .ssh directory.

For this example I have created one user rsync_user which will connect to the primary server and sync the files.

I will run rsync command on the DR site (Say IP 192.168.10.76), it will connect the Primary Site (Say IP 192.168.10.75) and sync the local directory of the DR site with the remote directory of the Primary site.

 

Steps to be done:

On the computer DR site (Say IP 192.168.10.76)

So on the DR site, login as the user that will connect the primary, in my case I will use my rsync_user user for this purpose.

Now run the command ssh-keygen to generate the keys.

$ ssh-keygen -t rsa

This will create the .ssh directory inside the home directory of the user (if already not present). When this command asks for passphrase, just press enter, do not enter any passphrase.

The above command will create a private key and a public key.

id_rsa is the private key and check the permission for this key. If the permission is not 600 then change the permission.

chmod 600 id_rsa

Now copy the id_rsa.pub file to the Primary Site (this is where I will connect using rsync).

 

On the computer Primary site (Say IP 192.168.10.75)

I have copied the id_rsa.pub file to the home directory of the rsync_user.

If .ssh folder is not present, then create it. Then add the id_rsa.pub key of the DR server into the authorized_keys file. If the file is not present then it will be created by the command cat id_rsa.pub >> .ssh/authorized_keys 

 

That’s it, we are set to connect through rsync/ssh without entering password.

On the DR server, I will run the rsync/ssh command. Before scripting, manually run the command once, so that the primary site is added to the .ssh/known_hosts file.

Now run the rsync command on the DR site.

Also we can connect ssh sessions without entering password.

All set, now I can add the shell script to sync my site’s folder. The sample script is (filename: /home/rsync_user/scripts/files_sync.sh):

echo "File sync started at" >> /home/rsync_user/scripts/files_sync.log
date >> /home/rsync_user/scripts/files_sync.log

rsync -rtvupgo --delete  rsync_user@192.168.10.75:/var/www/ /var/www/ >> /home/rsync_user/scripts/files_sync.log

echo "File sync completed" >> /home/rsync_user/scripts/files_sync.log
echo "" >> /home/rsync_user/scripts/files_sync.log

Create a cron job to do the sync say in every 5 minutes:

Run

crontab –e

and add the following line:

*/5 * * * *  sh /home/rsync_user/scripts/files_sync.sh

We are ready with our requirement

Linux LVM Explained

 

“Hey guys…. I heard a lot about LVM….. Please tell me what it is?”

• LVM stands for Logical Volume Manager (or Logical Volume Management).
• Logical Volume Manager allows for a layer of abstraction between your operating system and the disks/partitions it uses.
• It is a system of managing logical volumes, or file systems.
• With LVM, disks and partitions can be abstracted to contain multiple disks and partitions into one device.

 

“Hmmmm…. looks exciting….. Now tell me why should I use LVM……Tell me some advantages of using LVM….”

• Most operations can be done on the fly, while the system is running.
• LVM can expand a partition while it is mounted, if the file system used on it also supports that ( like the usual ext3/4 ).
• LVM can move a partition while it is in use, and will not corrupt your data if it is interrupted.
• It allows to take snapshot at any moment, even while the system is running.

 

“Well, you forced me to think….. So how LVM is different from traditional disk management?”

Traditional Disk Management

• Operating system looks for what disks are available (/dev/sda, /dev/sdb, etc.).
• Then looks at what partitions are available on those disks (/dev/sda1, /dev/sda2, etc.).

 

LVM Disk Management

• A physical disk is divided into one or more physical volumes.
• Physical Volumes corresponds to physical disk partitions.
• Logical volume groups are created by combining Physical Volumes.
• A logical volume group is a collection of physical volumes from which a logical volume can be created.
• Logical volumes correspond to partitions and they hold a file system.
• A logical volume lives in a volume group that is made up of one or more physical volumes.

 

“Ooppssss…. that was too much technical for me… but how I can manage LVM?”

There are adequate commands to manage LVM. Each LVM command is preceded by one of the following:

• Physical Volume = pv
• Volume Group = vg
• Logical Volume = lv

To get LVM information you can use

• The s (pvs, lvs, vgs) command:
• The display (pvdisplay, lvdisplay, vgdisplay) command:

 

“Things are going over my head….. Show me some practical stuffs……. Please do some LVM stuffs for me….. Okey, I will give you some scenarios…. and you have to do it for me….”

 

---

“Howdy Techies….Here I am to destroy your sleep…..

Add three new disks to my Linux machine…..

Create partitions on these three disks….”

 

So for this blog post…. I will use Oracle Virtual Box VM and Ubuntu Linux 12.04

I am adding three disks to my Ubuntu VM.

Now my VM has total 4 disks.

Start the VM after adding the disks.

I am going to use the fdisk command to check the newly added disks. We can see the three added disks in the below screen shot:

 

Partitioning 1st disk

We will create two primary partitions on the 1st disk /dev/sdb.

Creating 1st Partition of 600MB on disk /dev/sdb.

Change the partition type of the 1st partition to Linux LVM.

t = Change partition type
8e = Hexa decimal code for Linux LVM

Creating 2nd Partition of remaining space on disk /dev/sdb.

Writing the changes to the disk and making it permanent.

Checking the partitions on this disk.

 

Partitioning 2nd disk

We will create only a single primary partition on the 2nd disk /dev/sdc.

Changing partition type.

 

“Our disk partitioning is over!!!!!!

Partitioning is time consuming…

I am tired…. Show me something new so that I do not sleep…..

Okey…. Show me LVM Stuff…..”

 

Install LVM2

# apt-get install lvm2

Now we have 3 partitions of type Linux LVM.

1./dev/sdb1 600MB on 1st Disk

2./deb/sdb2 400MB on 1st Disk

3./dev/sdc1 1GB on 2nd Disk

We are going to create two physical volumes from the first two partitions of the above list.

Using these two physical volumes we will create a volume group.

After that we will create a logical volume of size 800MB.

Then we will format the logical volume and mount and use it.

 

Creating physical volumes

Use pvcreate command to create a LVM physical volume on the partitions.

 

Creating volume group

Our physical volumes are created, we need to create the volume group.

#vgcreate Volume_Group_name List_of_Physical_Volumes

 

Creating logical volume

To create the logical volume

# lvcreate -L Size -n Name_of_the_new_lv volume_group_name

Format and Mount the Logical Volume

Now we can see our new LVM logical volume mounted on /lvm_training

Copy some data into this volume.

 

“Easy enough!!!!!

Show some real use of LVM….”

“I am copying my personal stuffs in /lvm_training folder.

I have already consumed almost 500MB, just 262MB free space is remaining here.

I need to copy a movie of size 700MB here.

Urgently give me 500MB more space, I don’t have time, I am currently working with the files in /lvm_training folder.

I can’t unmount /lvm_training, I can’t shutdown the machine…… but I need more space here!!!!!!”

 

Adding more space

Right now our volume group vgtraining has size 1016MB, out of which 800MB is already allocated.
We need 500MB more space.
So we will add the /dev/sdc1 partition (of the 2nd disk) to this volume group, to increase its size.

Create a new Physical Volume from /dev/sdc1 partition.

Add the new Physical Volume to the Volume Group vgtraining. After adding the new Physical Volume we can see that the size of the volume group has increased.

Extend the logical volume:

We can specify the size of the logical volume in two ways:
-L+500M = Add 500MB more space to this volume.
-L1400M = Extend the volume size to 1400MB.

So I am going to add 500MB to the logical volume lvtraining1

Lets check the disk space utilization now.

 

 

“Oppss….. still our df command is showing space utilization as it was before.

We have added 500MB space to the volume group.

Where the new space has gone????????”

Extend file system

After extending the Logical Volume we need to expand the filesystem to use the new space.

For ext3 or ext4 filesystem run:

Now we can see the extended size of this logical volume.

 

“That was cool…..

But the movie I copied was only 300MB.

Right now I have 435MB free in /lvm_training.

You IT guys are charging me too much $/MB of storage.

So I have decided to release 250MB of free space, so that I can save few $ .”

 

Shrinking space

We have to unmount the partition to reduce the space from the partition as resize2fs do not support on-line shrinking.

Disk space utilization before shrinking

Unmount the partition

Reduce the size of the Logical Volume

-L : Specify the size.

-r : It will run the resize2fs and the filesystem checks.

Note: If you run the above command on a mounted partition, you may severely damage the file system in that partition.

Mount the partition again

Check disk space utilization again
We can now see that disk space has been reduced.

 

“What will happen if I run the previous command on a mounted partition?

Suppose I run the command to reduce space from the logical volume, but the amount to reduced I specified is more than the free space available on this partition.”

 

Running the previous command on a mounted partition:

It may damage the file system in that partition.

Running the previous command with size parameter more than the free space available in that partition:

The command will fail.

 

“That was quite impressive.

But guys… I am finding the disk speed quite low, can you give me a faster disk??????????

Again I am very busy… so I can’t give you any down time….”

 

Moving Partitions

We have added third disk (/dev/sdd) of 2GB size while adding the disks at the beginning of this hands on (we are assuming that this is a faster disk ).

Now we will add this disk to the volume group vgtraining and will move all the data of the disks (/dev/sdb and /dev/sdc) to this new disk.

We will remove the disks /dev/sdb and /dev/sdc

Create partition on /dev/sdd

Change partition type and write the changes.

Create Physical Volume.

Add this physical volume to the Volume group vgtraining.

Check the utilizations of all the physical volumes.

We can see that /dev/sdd1 is free right now.

First we will move the content of /dev/sdb1

If we run pvmove /dev/sdb1 without specifying the destination Physical Volume, then all the logical extents of any logical volumes on /dev/sdb1 will be moved elsewhere in the volume group.

Also if we run pvmove –n logical_vol /dev/sdb1 then Logical Volume logical_vol on the /dev/sda1 Physical Volume will be moved.

Removing Partitions

All the logical volumes are moved from the disks /dev/sdb and /dev/sdc.

Now we can remove all the partitions of /dev/sdb and /dev/sdc from the volume group.

Removing all the Physical volumes of /dev/sdb and /dev/sdc

New we will remove all the physical volumes

 

“Hey Guys you are doing wonderful jobs….

But my needs are ever-growing…..

Meet my folks…..

They are quite impressed with my collections of music and songs that I stored in my /lvm_training folder…. They need a copy of that folder…. Hey don’t think my requirement as a simple one….

I will keep updating that folder at any moment of time… but I want to give them the current content (content at this moment of time)…. While you are copying the files I may be updating/adding files in this folder…. But my folks should not get the new/updated contents.”

Snapshot

• The LVM allows you to take a snapshot of any logical volume.
• When you create a snapshot, you create a new Logical Volume to act as a clone of the original Logical Volume.
• If we make some changes to the original volume, the changed blocks are copied to the snapshot volume before they are changed.
• One of the coolest things about LVM snapshots is your file system is never taken offline.
• To create a snapshot we need to create a new logical volume with enough free space to hold any new information that will be written to the logical volume while we make a backup.

 

Create New Snapshot

Now it's time to create the snapshot of the /dev/vgtraining/lvtraining1

We can see the new logical volume for the snapshot

 

“I am updating contents of this folder now…..

I think I don’t need problem11.csv file…..I am going to delete this file…..”

“But you have to give my folks the previous contents of this folder.

Copy the contents to /tmp folder…. from there my folks will download it.”

Mount snapshot and copy

Once copying is done, we can unmount the snapshot volume.

 

“I am so fool…..

I have updated some source code files in the folder /lvm_training and one application depends on those files stopped running.

My boss yet to know about it…. If he learns about it… probably he will send me back to Africa…..

God help me…. Wish I could go back to the time when I asked you to take snapshot….”

 

Reverting/Merging To The Snapshot's State

• Once you have a snapshot available, you can change the original logical volume to the state of that snapshot.
• Unmount that logical volume if you want to see the change to happen immediately.
• If we run merging without unmounting the logical volume, then changes will take effect after next reboot.

Mount the Logical Volume

I can see the deleted file problem11.csv

 

“Thanks a ton….You saved me……..

LVM is cooollllllllllllllllllllll like me”