Page 1 of 1

RESOLVED: Crashplan and inotify watches

Posted: Fri Jul 10, 2015 2:47 pm
by jaybea
I was reviewing my logs when I spotted the ntpd service being started every minute or two. Digging into this further, I discovered that restarting ntpd manually (sudo systemctl restart ntpd.service) resulted in a "Error: Not enough space on device" message.

This got me worried, but checking drive space, I found that all the drives had plenty of space, and there were no shortage of inodes (a common reason for the message). I rebooted (the shutdown command also resulted in a "Not enough space on device" message, but it did restart), but this did not solve the problem. I then decided to watch the log (tail -f /var/log/messages) and spotted the following message:

Code: Select all

tail: inotify resources exhausted tail: inotify cannot be used, reverting to polling
A clue! It turns out that Crashplan needs a lot of inotify watches, and the default of 8192 had been used on my system, which resulted in the various "Not enough space on device" error messages, which were actually generated by inotify and nothing to do with ntpd or disk space.

There are some instructions for increasing the number of inotify watches on your system on the Crashplan website

Update: I spoke too soon - the ntpd service is still continually restarting (once every minute or two). No more "Not enough space messages", though!

Re: Crashplan and inotify watches

Posted: Fri Jul 10, 2015 4:47 pm
by bigfoot65
I have never heard of this happening.

So will increasing it more help? I am not a big fan of CrashPlan and it often causes problems with Greyhole.

Wonder if it's related to Greyhole Inodes issue.

Re: Crashplan and inotify watches

Posted: Sat Jul 11, 2015 1:22 am
by jaybea
So will increasing it more help? I am not a big fan of CrashPlan and it often causes problems with Greyhole.

Wonder if it's related to Greyhole Inodes issue.
It has got rid of the error messages, and it appears to be a known problem, and it is the same as the Greyhole issue, and the same solution should work (increasing the number of inodes available to the system). This does have an implication on system resources, so the number of inodes you make available needs to reflect system resources.

I have not resolved the NTPD issue yet - I need to monitor the process so see what is going on, but I wonder whether it is an issue with monit.

Re: Crashplan and inotify watches

Posted: Sat Jul 11, 2015 5:37 am
by bigfoot65
Do you have all the OS updates applied? Did you install ntpd or was it installed from the Network Time applicaiton.

Re: Crashplan and inotify watches

Posted: Sun Jul 12, 2015 5:05 am
by jaybea
Do you have all the OS updates applied? Did you install ntpd or was it installed from the Network Time applicaiton.
I think that I have found the problem. I had the Network Time application installed, but it restarted at the end of January following an update and appears to have been running without a pid file since. Monit, which checks for the presence of the pid file, therefore thought it was not running and once every minute, restarted it. I have now reinstalled the Network Time app, and it is now running with a pid file and monit is happy again.

Re: Crashplan and inotify watches

Posted: Sun Jul 12, 2015 7:29 am
by bigfoot65
Great to hear. Will consider this one resolved.