Linux Out of Memory Process Killer

Linux OS has an Killer….. Oooopppssss…. don’t afraid…. its just the "Out of Memory" killer facility which kills running processes when the system runs out of free memory. When the Linux system runs out of memory then the kernel starts killing processes in order to stay operational. The Linux kernel uses a mechanism called Out Of Memory Killer (or OOM Killer) for recovering memory on the system and overcome memory exhaustion.

In one of my server running LAMP stack sometimes MySQL server was getting terminated abruptly. Actually MySQL was getting killed by the OOM killer. MySQL memory pools were optimized but actually the server physically had low memory and there was no possibility of increasing memory of the server. I could afford other processes (like Apache) getting killed but have to prevent MySQL database server from getting killed.

Normally Linux OOM killer treats all processes equally, but there is a way to control the behavior of OOM Killer. Each Linux process has a OOM score assigned to it. Whenever system is about to run out of memory, OOM killer terminates the program with the highest score.

One way is to adjust the value of the file /proc/[process_id]/oom_adj (since Linux kernel 2.6.11). The valid range is –16 (very unlikely to be killed by the OOM killer) to 15 (very likely to be killed by the OOM killer) and a value of –17 exempts a process entirely from the OOM killer.

So we can do as root user:
# echo –17 > /proc/MySQL_Process_ID/oom_adj
to keep MySQL process out of reach of the OOM killer.

Since Linux 2.6.36, use of the file /proc/[process_id]/oom_adj  is deprecated in favor of the file /proc/[process_id]/oom_score_adj
The range of values which oom_score_adj accepts is from integer -999 (very unlikely to be killed by the OOM killer) up to 1000 (very likely to be killed by the OOM killer) and a value of –1000 exempts a process entirely from the OOM killer.

So in this case we have to set:

# echo -1000 > /proc/MySQL_Process_ID/oom_score_adj

to prevent MySQL getting killed.

But above two techniques are temporary, whenever we restart MySQL or 

the Server the Process ID of MySQL process changes and again we have to run the above command.

To permanently exempt MySQL from getting killed, we can edit the MySQL service’s upstart script file 

/etc/init/mysql.conf and add the parameter 
oom score 

The value of this parameter can be an integer ranging -999 (very unlikely to be killed by the OOM killer) to 1000 (very likely to be killed by the OOM killer). It may also have a special value never which instructs the OOM killer to ignore this process entirely.


So for my MySQL database server running on Ubuntu 12.04, I edited the upstart script /etc/init/mysql.conf and added the line:

oom score never

After that restart MySQL service and its done :) .

Lets check the values that /proc/MySQL_Process_ID/oom_score_adj and /proc/MySQL_Process_ID/oom_adj files have after setting oom score never

Yeppp… it is as expected :) :) :)