Pranab's scrapbook

Monday, 22 June 2015

Running node.js application with Passenger and Nginx

We can use Phusion Passenger to deploy our node.js application on production, it takes out many complications that come while deploying or running Node.js application. Using Passenger is very easy and some of the benefits that we get from using Passenger are:
1) We have nginx to serve the static contents, so our app can concentrate of serving our main purpose.
2) Ngnix also protects our application from some kind of attacks.
3) Automatic restart of our application on system reboots.
4) We can run multiple Node.js applications on a single server easily.
5) Phusion Passenger can spawn more Node.js processes depending on the load etc.
6) Automatic starts a new process for a failed process.

First we are going to install node.js
Note: For this demo I used Ubuntu 14.04

Installing node.js

# curl -sL https://deb.nodesource.com/setup | sudo bash -
Now run apt-get to install node.js
# apt-get install nodejs

Installing Ngnix

# apt-get install ngnix

Installing Passenger

Install passenger PGP key.
# apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 561F9B9CAC40B2F7
We need to add HTTPS support for apt as passenger apt repository is hosted on an HTTPS server.
# apt-get install apt-transport-https ca-certificates
Create a file /etc/apt/sources.list.d/passenger.list and insert the following line:

# Ubuntu 14.04

deb https://oss-binaries.phusionpassenger.com/apt/passenger trusty main

Note: the above line is specifically for Ubuntu 14.04, if you are not using Ubuntu 14.04 then please refer Passenger document to get the URL for your distribution.

After adding the new apt source, run apt-get update followed by apt-get install

# apt-get update

# apt-get install nginx-extras passenger

Once passenger installation is done, edit /etc/nginx/nginx.conf and uncomment the following lines:

passenger_root /usr/lib/ruby/vendor_ruby/phusion_passenger/locations.ini;
passenger_ruby /usr/bin/passenger_free_ruby;

You can verify the passenger root is correct or not by the command:

# /usr/bin/passenger-config --root

Installation part is over, not I am going to deploy one application to test the new setup.

For this demo I wrote a small app test.js

var http = require("http");
function onRequest(request, response){
var body = '<html>'+
'<head>'+
'<meta http-equiv="Content-Type" content="text/html; '+
'charset=UTF-8" />'+
'</head>'+
'<body>'+
'<h1>'+
'Hi I am SuperMan'+
'</h1>'+
'<img src=monkey1.gif>'+
'</body>'+
'</html>';
response.writeHead(200, {"Content-Type" : "text/html"});
response.write(body);
response.end();
}
http.createServer(onRequest).listen(9080);

console.log("Server has started.");

As suggested in the passenger documentation https://github.com/phusion/passenger/wiki/Phusion-Passenger%3A-Node.js-tutorial , I have created the application directory structure:

/apps

  +-- test.js

  +-- public/

  +-- tmp/

/apps is the root directory of my application

test.js file is the entry point of my application. Passenger will load test.js to start my application.

public : This directory contains static file, so files placed in this folder will directly serve by Nginx, request will not be forwarded to the application. I copied one image file monkey1.txt in this directory.

tmp : We can create restart.txt file in this directory to restart the application on next request. Also this directory can be used by the application also.

In this example I am going to run my Node.js application as my default site. So I am editing the file /etc/nginx/sites-available/default and my file looks like:

server {
listen 80 default_server;
listen [::]:80 default_server;
index index.html index.htm index.nginx-debian.html;
server_name _;
location / {
# First attempt to serve request as file, then
# as directory, then fall back to displaying a 404.
try_files $uri $uri/ =404;
}
passenger_app_root /apps;
passenger_enabled on;
passenger_app_type node;
passenger_startup_file test.js;

#This is where my static files will go
root /apps/public;
}

All set, restart nginx and we are ready.

# service nginx restart

My superb website is up and running Smile

Friday, 19 June 2015

Using PM2 process manager for node.js

PM2 (Process Manager 2) https://github.com/Unitech/PM2 is a process manager for Node.js applications. Some of its features are:
· PM2 allows us to run our application as multiple processes, without having to modify our application.
· It has built-in load balancing capabilities for running node.js application in cluster.
· It keeps the application up by restarting the application if it crashes.
· It can start the node application as a service when we restart the server.

Note: For this post I used Ubuntu 14.04

Installing node.js

Run the below command as root user or use sudo
# curl -sL https://deb.nodesource.com/setup | sudo bash -

Now run apt-get to install node.js
# apt-get install nodejs

Update 22/07/2015:
******************************************************

Suppose you want to install a different version of nodejs (at the time of writing this post the default setup was 0.10), for example suppose if I want 0.12, then go to the NodeJS github link:
https://github.com/nodesource/distributions/tree/master/deb

There you will get setup scripts for different versions of NodeJS, for my requirement of version 0.12 the script is setup_0.12, from the script get the link of the script and run the curl command:

curl -sL https://raw.githubusercontent.com/nodesource/distributions/master/deb/setup_0.12 | sudo bash -

After that run apt-get install nodejs command.

******************************************************

For testing cluster I created a small application test.js, this application shows the Process ID of the process which served the request:
For this application I need process module, install process module

My test.js file:
var http = require("http");
var process = require('process');

function onRequest(request, response){
response.writeHead(200, {"Content-Type" : "text/html"});
response.write('Request served by: ' + process.pid);
console.log(process.pid);
response.end();
}

http.createServer(onRequest).listen(8888);
console.log("Server has started.");

The sample output of this app:

Now I am going to install pm2 and run my test.js application using pm2.

Installing PM2:

$ sudo npm install pm2 -g

Running our application with pm2

Now we will start our test application using pm2 and will run multiple processes to form our cluster.
$ pm2 start test.js --name "testapp" -i 0
Options:
--name : This will specify a name for our application. This name can be used in other pm2 commands.
-i : Start the application in cluster mode, and the number of processes to run. A value of 0 informs pm2 to start as many worker processes as you have CPU cores.

So in our case we have 2 core CPU, so pm2 started 2 processes for our application.

Monitoring our application:

We can use the following pm2 commands to monitor our application:
Get list of applications monitored by pm2:
$ pm2 list

To get more details about an application:
$ pm2 show testapp
This will return details of all the processes running for that application. To get more details about a process, we can use the pm2 id for that process
$ pm2 show 1
Same output we can get using the pm2 desc command.

Monitoring the processes with CPU and RAM usage:
$ pm2 monit

The blue bar shows CPU usage and red one shows RAM usage for a particular process.

Checking logs of all the monitored processes:

$ pm2 logs
pm2 logs command shows tail command kind of output from all the log files. Log files for each individual processes are generated in PM2_PROCESS_USER_HOME_DIRECTORY/.pm2/logs directory.
So in my case /home/pranabs/.pm2/logs

To clean up the log files:
$ pm2 flush

To check if the cluster is working fine, I open my testapp in browser which displays the PID of the worker process which served the request.

If I keep refreshing the page, I could see different PIDs. If I check the displayed PIDs, I could find these PIDs in pm2 list command. This means that the user requests are being served from different processes which indicates some sort of load balancing is working in our cluster.

Restarting application:

pm2 reload <APP_NAME/all>
To restart all the monitored applications:
$pm2 reload all
To restart a particular application:
$pm2 reload testapp

Stopping application:

We can stop particular process/application or all applications
$ pm2 stop testapp

To remove application from pm2:

We can delete particular process/application or all applications from pm2
$ pm2 delete testapp

Checking auto start of failed process:

pm2 automatically starts a failed process. To verify that we will forcefully kill one of the processes and check whether pm2 starts another process for the failed one. Here I killed the process 5910, and we could see in the below screenshot that pm2 had started another process with ID 5939.

Starting applications automatically at server boot time

We can run the command pm2 startup to create the init script and deploy that init script to run at system startup.
To auto detect the platform just run pm2 startup:
$ pm2 startup
Also we can specify the platform with the startup command:
$ pm2 startup [platform]
The available options for platform are ubuntu, centos, redhat, gentoo, system, darwin and amazon
As I am using Ubuntu, so I am specifying ubuntu
$ pm2 startup ubuntu
This command will give us a command to run as root, which will actually deploy the init script at system startup.
sudo env PATH=$PATH:/usr/bin pm2 startup ubuntu -u pranabs

If we want to run the application as root, then we can directly run the pm2 startup command as root and it will deploy the necessary startup scripts.

Next run pm2 save command so that pm2 saves the currently monitored processes. The saved processes will be automatically started by pm2 when the system boots.
$ pm2 save

PM2 is very convenient tool for running node.js applications and the clustering is a very powerful feature. So try and enjoy pm2 Smile

Wednesday, 7 January 2015

Linux Out of Memory Process Killer

Linux OS has an Killer….. Oooopppssss…. don’t afraid…. its just the "Out of Memory" killer facility which kills running processes when the system runs out of free memory. When the Linux system runs out of memory then the kernel starts killing processes in order to stay operational. The Linux kernel uses a mechanism called Out Of Memory Killer (or OOM Killer) for recovering memory on the system and overcome memory exhaustion.

In one of my server running LAMP stack sometimes MySQL server was getting terminated abruptly. Actually MySQL was getting killed by the OOM killer. MySQL memory pools were optimized but actually the server physically had low memory and there was no possibility of increasing memory of the server. I could afford other processes (like Apache) getting killed but have to prevent MySQL database server from getting killed.

Normally Linux OOM killer treats all processes equally, but there is a way to control the behavior of OOM Killer. Each Linux process has a OOM score assigned to it. Whenever system is about to run out of memory, OOM killer terminates the program with the highest score.

One way is to adjust the value of the file /proc/[process_id]/oom_adj (since Linux kernel 2.6.11). The valid range is –16 (very unlikely to be killed by the OOM killer) to 15 (very likely to be killed by the OOM killer) and a value of –17 exempts a process entirely from the OOM killer.

So we can do as root user:
# echo –17 > /proc/MySQL_Process_ID/oom_adj
to keep MySQL process out of reach of the OOM killer.

Since Linux 2.6.36, use of the file /proc/[process_id]/oom_adj is deprecated in favor of the file /proc/[process_id]/oom_score_adj
The range of values which oom_score_adj accepts is from integer -999 (very unlikely to be killed by the OOM killer) up to 1000 (very likely to be killed by the OOM killer) and a value of –1000 exempts a process entirely from the OOM killer.

So in this case we have to set:

# echo -1000 > /proc/MySQL_Process_ID/oom_score_adj

to prevent MySQL getting killed.

But above two techniques are temporary, whenever we restart MySQL or

the Server the Process ID of MySQL process changes and again we have to run the above command.

To permanently exempt MySQL from getting killed, we can edit the MySQL service’s upstart script file

/etc/init/mysql.conf and add the parameter 
oom score

The value of this parameter can be an integer ranging -999 (very unlikely to be killed by the OOM killer) to 1000 (very likely to be killed by the OOM killer). It may also have a special value never which instructs the OOM killer to ignore this process entirely.

So for my MySQL database server running on Ubuntu 12.04, I edited the upstart script /etc/init/mysql.conf and added the line:

oom score never

After that restart MySQL service and its done :) .

Lets check the values that /proc/MySQL_Process_ID/oom_score_adj and /proc/MySQL_Process_ID/oom_adj files have after setting oom score never

Yeppp… it is as expected :) :) :)

Friday, 2 May 2014

Optimizing MySQL queries with memory tables

In this query optimization tip, I will show you how we can make our queries faster using MySQL’s memory storage engine.

I have some report queries in my call center application (using which we capture the calls that we received in our call center).

The main table to be used was callflow:

I have to calculate some stats from this table, like the total duration of the call.

One of the report query was:

select date(receivedon), sum(TIME_TO_SEC(TIMEDIFF(finishedon,receivedon))) Total, count(rcv_call_id) No_of_calls,
max(TIME_TO_SEC(TIMEDIFF(finishedon,receivedon))) maximum ,
min(TIME_TO_SEC(TIMEDIFF(finishedon,receivedon))) Minimum
from
(select FkCallID rcv_call_id, receivedon from callflow where FlowType='R' and
ReceivedOn >= '2014-01-01 00:59:03' and
ReceivedOn <= '2014-01-31 00:59:03') rcv,
(select FkCallID clsd_call_id, finishedon from callflow where FlowType='C' and
FinishedOn >= '2014-01-01 00:59:03' and
FinishedOn <= '2014-01-31 00:59:03') clsd where rcv_call_id=clsd_call_id group by date(receivedon);

This query returns total number of calls for a day, total time in seconds for the calls, max time taken for a call and min time taken for a call.

This query was taking around 30 seconds to execute which is too much.

So I decided to use MySQL’s memory tables for above query. The above query has two derived tables and I am going to use memory tables for those two derived tables.

1) First I will create the memory tables for the derived tables and going to add index on the column which will be used in where condition.

CREATE TABLE rcv ENGINE=MEMORY
SELECT FkCallID rcv_call_id, receivedon from callflow where FlowType='R' and
ReceivedOn >= '2014-01-01 00:59:03' and
ReceivedOn <= '2014-01-31 00:59:03' ;

ALTER TABLE rcv ADD INDEX (rcv_call_id);

CREATE TABLE clsd ENGINE=MEMORY select FkCallID clsd_call_id, finishedon from callflow where FlowType='C' and
FinishedOn >= '2014-01-01 00:59:03' and
FinishedOn <= '2014-01-31 00:59:03';

ALTER TABLE clsd ADD INDEX (clsd_call_id);

2) Using these memory tables, I will rewrite the query:

3) After running the query and getting the result, I will drop the temporary memory tables:

drop table rcv;
drop table clsd;

My query after using the memory tables gave results in 87 ms which is lightning fast compared to 30 secs it was taking previously with derived tables.

But remember to adjust the max_heap_table_size system variable, as this value restrict the maximum size of memory tables. Default is 16MB, so adjust it as per your need to take the benefit of memory tables.

Monday, 21 April 2014

Phabricator & Code Auditing

We wanted to implement code quality auditing in our software development lifecycle. We came across awesome Phabricator platform which makes code auditing much easier.

In this document I will show, how to install Phabricator in Ubuntu Linux and configuring it to Audit codes for some particular user(s) in an external SVN repository.

Installation & Setup:

We use Ubuntu 12.04 in our production and there is a script for Ubuntu installation. You can download the script from the link: http://www.phabricator.com/rsrc/install/install_ubuntu.sh

As we are using Ubuntu 12.04, so I changed the following line in the install_ubuntu.sh script:

Copy the install_ubuntu.sh script to the folder where you want to install Phabricator. I want to install Phabricator in /usr/local/phabricator, so I copied the script in /usr/local

Make the script executable and run the script as root user (or using sudo, I hate sudo and prefer to go to the root login)

# chmod +x install_ubuntu.sh

# ./install_ubuntu.sh

Change ownership of phabricator folder, so that Apache webserver has access to it

# chown -R www-data:www-data /usr/local/phabricator/

Now I am going to create a virtual host for my phabricator site

# pico /etc/apache2/sites-available/phabricator

# a2ensite phabricator

# service apache2 reload

If required change the MySQL database configuration for Phabricator

# /usr/local/phabricator/bin/config set mysql.user mysql_username

# /usr/local/phabricator/bin/config set mysql.pass mysql_password

# /usr/local/phabricator/bin/config set mysql.host mysql_host

# /usr/local/phabricator/bin/storage upgrade

Open the new Phabricator site, and create the initial administrator account.

Configure strict-mode for MySQL:

Open /etc/mysql/my.cnf file and add the following line under [mysqld] section:

sql-mode = STRICT_ALL_TABLES

After that restart MySQL

# service mysql restart

Disable apc.stat in the /etc/php5/conf.d/apc.ini by adding the following line:

apc.stat=0

Add timezone to PHP’s config file:

/etc/php5/apache2/php.ini

date.timezone = Asia/Calcutta

(Set your correct timezone, for me it is Asia/Calcutta)

If you are planning to user LDAP/Active Directory authentication with your Phabricator instance, you have to install PHP LDAP module.

# apt-get install php5-ldap

Restart Apache

# service apache2 restart

Also we have to install subversion

# apt-get install subversion

Create a local repository directory:
# mkdir -p /data/repo

Edit the repository.default-local-path key to the new local repository directory.

Go to Config -> Current Settings -> repository.default-local-path

Set the Base URI of Phabricator install:

# /usr/local/phabricator/bin/config set phabricator.base-uri 'http://phabricator.mkcl.org/'

Also we have to start Phabricator daemons

# /usr/local/phabricator/bin/phd start

Configuring external SMTP

We need to use external SMTP server for sending mails, for that we will use PHPMailer.

Go to Config

Click on PHPMailer

In PHPMailer configuration, enter your external SMTP server’s details.

Now I am going to change the Mail Settings. Go to Config -> Mail

Edit metamta.mail-adapter, here select PhabricatorMailImplementationPHPMailerAdapter

Configuring Active Directory Authentication:

We are going to use Active Directory authentication for Phabricator user login, so in this section I will show Active Directory integration.

Click on Add Authentication Provider

Select LDAP from the Provider list.

Enter LDAP hostname, LDAP Port and Base Distinguished Name

I am binding to LDAP with users’ LDAP username and password (details of LDAP binding is available in Authentication Provider creation page)

User Login

In this example I will use two users, one is pranabs who is our code Auditor and second one is websafe who is our developer.

As we have configured LDAP authentication, so user can login with their Active Directory login. I will show the process for our first user pranabs.

After LDAP authentication is successful, some additional information is also required for that user (as I configured simple direct binding). After entering Email and Real Name click on the Register Phabricator Account button.

This registration request has to be approved by the admin, after that the user can login.

Also the user will get one Email in his/her registered Email ID for verifying the Email address that is entered at the time of registration.

Click on the link received in the Email and verify the Email address.

The Admin will receive Email for approving the new registered user.

Also the admin will see the count of the number of new users to approve in his/her login

Admin has to go to the link People -> Approval Queue and click on the Approve button to approve the new user.

Once the new account is approved, the user will receive Email alert that his/her account has been approved.

Configuring external SVN repository

We have few existing SVN repositories, so instead of using hosted repository of Phabricator I planned to use the external repositories.

To add the SVN repository, login as admin and go to the Repositories link under Administration section:

Click on Create New Repository

As we are going to use existing external repository, so I am selecting Import an Existing External Repository.

The type of our repository is Subversion, so I am selecting Subversion.

Give a name to the new repository, also give a Callsign. Callsign is a short unique identifier for the repository and mainly it will be used for repository related operations.

Next enter the root of the repository. I can access my repository using http, so I entered http link for my repository root.

If the repository needs authentication, we have to create Credential for the repository. Click on the Add Credential button.

Enter the required details and select the visibility and who can edit this credential.

Select the just added Credential and click on Continue button.

Select access policy for this repository.

All done, we can now start importing the repository data. Select Start Import Now and click on Save button.

The background daemon will start reading updates from the repository; we can see the status of import.

To view the repository commits and contents go to the Diffusion link

Here we can see the list of configured repositories (on which we have access permission)

Commit details:

We can see the details of a commit, including the file content.

Creating Audit Rule

As per our requirement, the commits are to be audited for a particular SVN repository. As I wrote earlier, for this example I have two users: pranabs (auditor) and websafe (developer). To send Audit requests for commits done by websafe to pranabs, we have to specify some rule or condition that will create the Audit action. For that I will use Herald rule (https://secure.phabricator.com/book/phabricator/article/herald/ ).

Go to Herald and click on Create Herald Rule link

Our requirement is for Commits, so I selected Commits.

I will create Object type rule as this rule will be for PranabTestRepo repository that we added for this example.

Now I have to enter the object name, in my case it is rPT (Callsign for my repository PranabTestRepo).

Next I have to enter the name for the Herald rule, also I have to create the rule triggering condition and the action that will be taken if the condition is satisfied.

So my condition is: if the Author is websafe, then the first action will be Trigger an Audit by pranabs and second action will be sending Email to pranabs and websafe about this Audit.

Once we created our rule, we can test the rule by going to Test Console

Here I entered one commit ID to test

Test Result: I can see my rule is successfully applied, but here no action is taken as this is a test run.

Auditing:

Suppose our developer websafe did some changes in the code and commit the changes.

Our Auditor pranabs will receive Email alert in his mailbox for the Audit request.

When auditor pranabs logins to Phabricator, he can see the Audit request.

Once Auditor opens the Audit request, details of that commit can be seen. Also Auditor can view the difference of the commit with the previous version of a file.