CopyDisable

Wednesday, 21 August 2013

Linux Glassfish Error: java.io.FileNotFoundException: (No space left on device)

Well this error is self explanatory, it is clear that there is no free space in the device (i.e. in my disk). I faced this issue in one of our development glassfish server running on Ubuntu 12.04. Glassfish server was not getting started or stopped properly and running very slow. When I checked logs I found this line:

WARNING|glassfish3.1.2|com.sun.enterprise.v3.server.SystemTasks|_ThreadID=10;_ThreadName=Thread-2;|Internal Error: java.io.FileNotFoundException: /app/glassfish/glassfish/domains/domain1/config/pid (No space left on device)|#]

cartoon

What the hell is this????? My disk space utilization monitoring is not giving me any alert, what may be the case?

When I run df command I could see that all the partitions are below the high disk space usage threshold mark. My Glassfish is deployed in /app partition, and this partition has 4.2GB free space, Thinking smile what may be the case?????

Untitled

Well may be that partition consumed all the inodes for that file system, lets check again:

This time I run the df command to display inode utilization:
image

Now I can see that /app partition’s file system had already used all the available inodes. As there is no free inode, so new files can not be created on that partition.

So where these large number of files came from? I have 15 web applications deployed on this development server, so I need to find where these files are residing.

So I drill down and as expected I found that most of these files are in the Glassfish domain’s generated folder.

image

I did some web searching if I can increase the number of inodes for an existing file system, but only solution I found was to recreate the file system with a largeer inode table Sad smile

I could stop the Glassfish domain, remove the files from glassfish’s generated folder, and start the domain. This will temporarily remove my problem, but as this server is hosting >= 15 web applications (also as it is a development server) so again number of files in Glassfish’s generated folder will grow and in near future same problem will come. So it is better to increase the number of inodes for this partition.

So I took down time, removed the files from Glassfish’s generated folder. Moved glassfish folder to a different partition.

Unmount the partition and create file system (in my case it is of type ext4) on this partition again with higher number of inodes.

image

Previously the /app partition had 655360 inodes, now I will double this number to
1310720 specifying the –N option of mkfs command.

image

Once file system is recreated with higher number of inodes, mount the pratition.
image

Copy the glassfish folder to this partition (my /app  partition) again and start glassfish.

It’s done………

Time to relax and have some Coffee cup .