Well this error is self explanatory, it is clear that there is no free space in the device (i.e. in my disk). I faced this issue in one of our development glassfish server running on Ubuntu 12.04. Glassfish server was not getting started or stopped properly and running very slow. When I checked logs I found this line:
WARNING|glassfish3.1.2|com.sun.enterprise.v3.server.SystemTasks|_ThreadID=10;_ThreadName=Thread-2;|Internal Error: java.io.FileNotFoundException: /app/glassfish/glassfish/domains/domain1/config/pid (No space left on device)|#]
What the hell is this????? My disk space utilization monitoring is not giving me any alert, what may be the case?
When I run df command I could see that all the partitions are below the high disk space usage threshold mark. My Glassfish is deployed in /app partition, and this partition has 4.2GB free space, 
Well may be that partition consumed all the inodes for that file system, lets check again:
This time I run the df command to display inode utilization:
Now I can see that /app partition’s file system had already used all the available inodes. As there is no free inode, so new files can not be created on that partition.
So where these large number of files came from? I have 15 web applications deployed on this development server, so I need to find where these files are residing.
So I drill down and as expected I found that most of these files are in the Glassfish domain’s generated folder.
I did some web searching if I can increase the number of inodes for an existing file system, but only solution I found was to recreate the file system with a largeer inode table 
I could stop the Glassfish domain, remove the files from glassfish’s generated folder, and start the domain. This will temporarily remove my problem, but as this server is hosting >= 15 web applications (also as it is a development server) so again number of files in Glassfish’s generated folder will grow and in near future same problem will come. So it is better to increase the number of inodes for this partition.
So I took down time, removed the files from Glassfish’s generated folder. Moved glassfish folder to a different partition.
Unmount the partition and create file system (in my case it is of type ext4) on this partition again with higher number of inodes.
Previously the /app partition had 655360 inodes, now I will double this number to
1310720 specifying the –N option of mkfs command.
Once file system is recreated with higher number of inodes, mount the pratition.
Copy the glassfish folder to this partition (my /app partition) again and start glassfish.
It’s done………
Time to relax and have some 
No comments:
Post a Comment