CopyDisable

Monday 26 August 2013

Ubuntu Password Expiration Policy

Suppose we want to enforce a password expiration policy in our Ubuntu Linux system, so that users have to change their passwords every 45 days. By default no password expiration policy is in place in Ubuntu, so user account’s password never expires. We can view the policy using chage command:

e.g. We are checking password expiration policy for the user admino
# chage -l admino

-l : show account aging information

image

I want to change the password expiration to 45 days so that the user is forced to change password every 45 days.

# chage -M 45 admino

-M MAX_DAYS  : set maximim number of days before password                         change to MAX_DAYS

After modifying the password expiration for the user admino, lets examine the aging information:

image

We can see that the policy has been enforced to the user’s account.

The /etc/shadow file contains user's password aging information. We may edit this file also to change the password policy.

image

Suppose we have a user test, and we want to set the number of days of warning before a password change is required to 10 days.

image

We can edit the /etc/shadow file and change the password warning period field for the test user to 10.

image

Now lets see the password aging information again:

image

We can see the change Smile.

The above things are can be done for existing users. But if we want deploy a default password expiration policy for all the new users that will be created in future. In that case we can edit the /etc/login.defs file and change the required parameters.

Say we want to deploy default maximum password expiration of 30 days and password expiration warning of 10 days when a new user is created.

open /etc/login.defs and edit the PASS_MAX_DAYS and PASS_WARN_AGE variables.

image

Friday 23 August 2013

Time spent report in redmine

We use Redmine (currently 2.1.6) for issue tracking system. Most of our users wanted to generate their time spent report from it. As redmine’s reports were not fulfilling our requirements, so I decided to write one small report app as per our requirement. I am Ruby on Rails illiterate Sad smile, and as the requirement was immediate, so I wrote the report in PHP. Just changed a little code of redmine to redirect one user to his/her report page.

To access this report one has to go to the My account link

image

I modified the source code of this file to add the View Time Spent Report on this page. To generate report click on this link.

image

 

Currently two types of reports can be generated, first is Show All Tasks. Using it you can see all your issue updates (wherever you had specified some time spent value) for a selected period.

clip_image001

To view details of an issue, click on the issue id.

clip_image002

The second report you can generate, is project wise time spent report. If you are working in multiple projects, this report can be useful for you.

clip_image003

clip_image004

Wednesday 21 August 2013

Linux Glassfish Error: java.io.FileNotFoundException: (No space left on device)

Well this error is self explanatory, it is clear that there is no free space in the device (i.e. in my disk). I faced this issue in one of our development glassfish server running on Ubuntu 12.04. Glassfish server was not getting started or stopped properly and running very slow. When I checked logs I found this line:

WARNING|glassfish3.1.2|com.sun.enterprise.v3.server.SystemTasks|_ThreadID=10;_ThreadName=Thread-2;|Internal Error: java.io.FileNotFoundException: /app/glassfish/glassfish/domains/domain1/config/pid (No space left on device)|#]

cartoon

What the hell is this????? My disk space utilization monitoring is not giving me any alert, what may be the case?

When I run df command I could see that all the partitions are below the high disk space usage threshold mark. My Glassfish is deployed in /app partition, and this partition has 4.2GB free space, Thinking smile what may be the case?????

Untitled

Well may be that partition consumed all the inodes for that file system, lets check again:

This time I run the df command to display inode utilization:
image

Now I can see that /app partition’s file system had already used all the available inodes. As there is no free inode, so new files can not be created on that partition.

So where these large number of files came from? I have 15 web applications deployed on this development server, so I need to find where these files are residing.

So I drill down and as expected I found that most of these files are in the Glassfish domain’s generated folder.

image

I did some web searching if I can increase the number of inodes for an existing file system, but only solution I found was to recreate the file system with a largeer inode table Sad smile

I could stop the Glassfish domain, remove the files from glassfish’s generated folder, and start the domain. This will temporarily remove my problem, but as this server is hosting >= 15 web applications (also as it is a development server) so again number of files in Glassfish’s generated folder will grow and in near future same problem will come. So it is better to increase the number of inodes for this partition.

So I took down time, removed the files from Glassfish’s generated folder. Moved glassfish folder to a different partition.

Unmount the partition and create file system (in my case it is of type ext4) on this partition again with higher number of inodes.

image

Previously the /app partition had 655360 inodes, now I will double this number to
1310720 specifying the –N option of mkfs command.

image

Once file system is recreated with higher number of inodes, mount the pratition.
image

Copy the glassfish folder to this partition (my /app  partition) again and start glassfish.

It’s done………

Time to relax and have some Coffee cup .

Friday 16 August 2013

Running sudo without password

 

I was writing a script which needs a normal user (part of sudo group) to stop and start one Ubuntu Service using service command. But problem was that if I need to run service command using sudo, I had to type the password of that normal user. This was killing the automation of the script. The way was to edit the /etc/sudoers configuration file.

I am showing the steps for a normal user stgsync

Step 1: Add the user to sudo group
usermod -a -G sudo stgsync

Step2: Now edit the /etc/sudoers file using the visudo command

Step3: Add the following line

%stgsync ALL=(ALL) NOPASSWD: /usr/sbin/service

This line will allow stgsync user to run service command without entering password.

For the details of this sudoers module and writing sudoers policy statements visit this page:

http://www.sudo.ws/sudoers.man.html

Step4: Save and exit from visudo

That’s it Smile, now the stgsync user can run service command without password:

e.g.
sudo service mysql stop
will not ask for password

Tuesday 13 August 2013

Automated MySQL database backup restoration

 

Introduction:

Mainly we use mysqldump to take backups of our MySQL database servers. Although mysqldump is very reliable but we need to make sure that backup should be restorable when it really matters. So there should be some backup verification process. As per this process we need to download latest database mysqldump backups from our MySQL database servers and restored it to some test server regularly after some days of interval. This activity is used to verify the integrity and reliability of the backups that are getting generated. Previously this activity was manual and it was taking human working hours to monitor and account/report the restoration activity. My task was to automate this activity and generate report of this activity for later analysis.

 

 

Platform:

We use MySQL 5.5 for our database engine on Ubuntu 12.04 64bit edition.

 

 

Technologies/Tools Used:

MySQL 5.5, PHP 5.3 scp, bzip2, shell scripting on Ubuntu 12.04

 

 

 

The Task:

It consists of two components:

1) A Shell script and a PHP script to do the restoration and statistics generation

2) A small reporting PHP-MySQL web application to generate reports from the gathered statistics.

Restoration and statistics generation

A normal OS user (stgsync) is created in all the database servers and in the backup testing server. The shell script will use this user to connect to a database server to download the latest backup using Linux’s builtin scp tool.clip_image003

After the backup file is downloaded, it is decompressed.

Before restoring the decompressed MySQL backup, MySQL database server in the Backup Testing Server will be cleaned up (removing all the existing data and log files).

MySQL server is cleaned up and restarted after that the decompressed mysqldump backup file is restored.

If restoration is successful, a PHP script will be called to generate statistics from the restored databases, these statistics are added to a centralized MySQL report database.

Restoration log will be mailed to the concerned people.
clip_image005

Backup Restoration Report Generation

When we open the home page of the report application, we can see when the last restoration took place for a particular MySQL database server.

clip_image007

In the above screenshot we can see last restoration information for the four configured database servers:

1) Restoration of the backup for the server IA6-MKCLSUPPORT-AS-01-P took place on 2013-08-12 and the name of the backup file is all_db_2013-08-12_04-00.bz2

2) Restoration of the backup for the server NMS-MKCLOS-DB-01-P took place on 2013-08-06 and the name of the backup file is all_db_2013-08-06_02-00.bz2

3) Restoration of the backup for the server NMS-PORTALS-AS-01-P took place on 2013-08-08 and the name of the backup file is all_db_2013-08-08_02-00.bz2

4) Restoration of the backup for the server SEW-SETS-DB-01-S took place on 2013-08-12 and the name of the backup file is all_db_2013-08-12_02-00.bz2

We can see the details of the latest backup restoration for a server, click on View Details

clip_image009

We can see the statistics of the restoration like how many table are restored, how many views are restored, how many procedures are restored etc.

clip_image010

Now to get more information about different restored items, say I need to check which tables are restored for a particular database, click on the number of tables restored.

clip_image011

I can get the table names and number of rows restored. Here the tables with 0 rows restored are shown in dark color.

clip_image013

To find which views are restored, click on the number of views for a database

clip_image014

clip_image015

Same way we can view the functions, procedures and events restored for a particular table.

Also we can get the detailed information by clicking View Details link

clip_image016

clip_image018

Now say we want to compare backup restorations reports for a particular database server, so that we are sure that backups are happening properly (by checking number of objects restored, number of tables and rows restored).

For that we have to find out when the previous backup restorations took place for a particular period

clip_image020

Select the database server name and select the backup restoration period (selecting a Start date and End date) and click on Go button.

For example we will find out when backup restoration took place for the server IA6-MKCLSUPPORT-AS-01-P from 1st of August 2013 and 13th of August 2013. After searching we could see that backup restoration took place 5 times.

clip_image022

Say we want to compare backup restoration that took place on 13-08-2013 and 12-08-2013. Click on View Details links for both backup restoration records. Here we can see how many different objects are restored for each database.

clip_image024

Now say we want to compare table restoration for the database redmine_ajitj, click on the number of tables restored for both restoration reports.

clip_image026

So we can see the number of rows restored for each table and now we can compare J. In this example we can see the latest backup restored lesser number of rows than the previous backup. There may be some issue or may be the application owner had cleaned up some data, so it needs some attention.

clip_image028

Also if required I can generate Table-Row report (restoration report of tables and number of rows for a particular server, database or table). For this type of report click on Table-Row Report link.

clip_image030

Here we can generate report as per our requirement. I will show you three possibilities.

1) We want Table and number of rows restoration report for a particular database server.
e.g. We will generate report for the server NMS-MKCLOS-DB-01-P
Select the Database server name, in Select Database Name list, leave it as --All Databases-- and in Select Table Name list keep --All-Tables-- . Select the report period by selecting Start Date and End Date.
clip_image032

Click on Go button to generate the report
clip_image034

2) We want to generate report for a particular database within a database server
e.g. We will generate report for the survey database in server NMS-MKCLOS-DB-01-P.
Select the Database server name NMS-MKCLOS-DB-01-P, in Select Database Name list select survey and in Select Table Name list select --All-Tables--. Select the report period by selecting Start Date and End Date.
clip_image036
Click on Go button to generate the report
clip_image038

3) In last report type, we can get restoration report for a particular table within a database.
e.g. We will generate report for the table survey_answer of the database survey in server NMS-MKCLOS-DB-01-P
Select the Database server name NMS-MKCLOS-DB-01-P, in Select Database Name list select survey and in Select Table Name list select survey_answer . Select the report period by selecting Start Date and End Date.
clip_image040
Click on Go button to generate the report
clip_image042

 

Conclusion:

This automation will save lots of human resource hour which was previously being wasted in database backup restoration, verification and reporting tasks. Also this will help us to find errors/issues in database backups and to find out inconsistent backups.