Run out of INodes

[dsmialek]

So my server has started having a some problems - some webapps won't run properly, vncserver won't start, yum can't update. After some digging around, I determine I have run out of "inodes" at /var. Being a linux novice, I have no idea what these are or how to resolve the problem. I manages to get things running again by deleting some files based on some help I got on the IRC channel, but the problem will surely return.

I have read that this can be a problem if I have a lot of small files. Problem is, I don't know how to find files I can safely delete. Also, it seems that /var was allocated a very small number of inodes for some reason. I'm not sure what I did so differently in the install that led me to this problem and apparently no one else.

Here is the output of the df and df -i commands I found. Notice all have disk space available and only /var is out of inodes. All others are not even close.

• [root@server ~]# df
  Filesystem 1K-blocks Used Available Use% Mounted on
  /dev/sdd3 10321208 3501140 6295780 36% /
  tmpfs 1899788 284 1899504 1% /dev/shm
  /dev/sdd1 247919 59411 175708 26% /boot
  /dev/sdd2 20642428 15999576 3594276 82% /home
  /dev/sdd7 200583196 4215408 186178736 3% /scratch
  /dev/sdd6 2064204 1191464 767884 61% /var
  /dev/sda1 1465004928 1212407756 252597172 83% /var/hda/files
  /dev/sdb1 976760000 723637852 253122148 75% /media/1TB SATA
  [root@server ~]# df -i
  Filesystem Inodes IUsed IFree IUse% Mounted on
  /dev/sdd3 655360 133799 521561 21% /
  tmpfs 474947 7 474940 1% /dev/shm
  /dev/sdd1 64000 44 63956 1% /boot
  /dev/sdd2 1310720 1524 1309196 1% /home
  /dev/sdd7 12738560 1024 12737536 1% /scratch
  /dev/sdd6 131072 131072 0 100% /var
  /dev/sda1 293027200 79324 292947876 1% /var/hda/files
  /dev/sdb1 253253220 26214 253227006 1% /media/1TB SATA

Any suggestions on how I can fix this?

[dsmialek]

I'm pretty sure I've found the problem.

/var/spool/clientmqueue is full of many, many small files. 90K and still counting. They are just text files and seem to be error messages from different apps. I've seen transmission and apache referenced.

From what I've read, this queue is for messages that are supposed to be emailed out. I don't have any email configured on the server so I guess they just sit in the queue forever.

I think I should be able to delete out all the files without any consequence, but now I've got to figure out how to stop the messages from queuing up.

[bigfoot65]

You can try to shut off sendmail. As root do the following in terminal window:

chkconfig sendmail off
service sendmail stop

That will stop if from starting on reboot and kill the current process. Hope this helps.

[NeverSimple]

I'm pretty sure I've found the problem.

/var/spool/clientmqueue is full of many, many small files. 90K and still counting.

I've checked the dir on my server; it holds far less files: 616. The server first became operational on 19 aug 2010.

From what I've read, this queue is for messages that are supposed to be emailed out.

That seems right. Al those files on my server are from greyhole:

Code: Select all

H??To: root H??Subject: fsck of Greyhole shares on localhost.localdomain H??X-PHP-Originating-Script: 0:greyhole

(not sure why i would want an e-mail about that, but maybe that's because greyhole is logging with -debug level, or something))

I've got to figure out how to stop the messages from queuing up.

Maybe you could tell from the subject line if there is 1 app that's trying to send so many messages (grep?). I think it would be best to disable them at the source, which probably isn't sendmail (?)


Richard