RESOLVED: Crashplan and inotify watches

jaybea
Posts: 51
Joined: Tue Dec 15, 2009 1:24 am

RESOLVED: Crashplan and inotify watches

Postby jaybea » Fri Jul 10, 2015 2:47 pm

I was reviewing my logs when I spotted the ntpd service being started every minute or two. Digging into this further, I discovered that restarting ntpd manually (sudo systemctl restart ntpd.service) resulted in a "Error: Not enough space on device" message.

This got me worried, but checking drive space, I found that all the drives had plenty of space, and there were no shortage of inodes (a common reason for the message). I rebooted (the shutdown command also resulted in a "Not enough space on device" message, but it did restart), but this did not solve the problem. I then decided to watch the log (tail -f /var/log/messages) and spotted the following message:

Code: Select all

tail: inotify resources exhausted tail: inotify cannot be used, reverting to polling
A clue! It turns out that Crashplan needs a lot of inotify watches, and the default of 8192 had been used on my system, which resulted in the various "Not enough space on device" error messages, which were actually generated by inotify and nothing to do with ntpd or disk space.

There are some instructions for increasing the number of inotify watches on your system on the Crashplan website

Update: I spoke too soon - the ntpd service is still continually restarting (once every minute or two). No more "Not enough space messages", though!

User avatar
bigfoot65
Project Manager
Posts: 10941
Joined: Mon May 25, 2009 4:31 pm

Re: Crashplan and inotify watches

Postby bigfoot65 » Fri Jul 10, 2015 4:47 pm

I have never heard of this happening.

So will increasing it more help? I am not a big fan of CrashPlan and it often causes problems with Greyhole.

Wonder if it's related to Greyhole Inodes issue.
ßîgƒσστ65
Applications Manager

jaybea
Posts: 51
Joined: Tue Dec 15, 2009 1:24 am

Re: Crashplan and inotify watches

Postby jaybea » Sat Jul 11, 2015 1:22 am

So will increasing it more help? I am not a big fan of CrashPlan and it often causes problems with Greyhole.

Wonder if it's related to Greyhole Inodes issue.
It has got rid of the error messages, and it appears to be a known problem, and it is the same as the Greyhole issue, and the same solution should work (increasing the number of inodes available to the system). This does have an implication on system resources, so the number of inodes you make available needs to reflect system resources.

I have not resolved the NTPD issue yet - I need to monitor the process so see what is going on, but I wonder whether it is an issue with monit.

User avatar
bigfoot65
Project Manager
Posts: 10941
Joined: Mon May 25, 2009 4:31 pm

Re: Crashplan and inotify watches

Postby bigfoot65 » Sat Jul 11, 2015 5:37 am

Do you have all the OS updates applied? Did you install ntpd or was it installed from the Network Time applicaiton.
ßîgƒσστ65
Applications Manager

jaybea
Posts: 51
Joined: Tue Dec 15, 2009 1:24 am

Re: Crashplan and inotify watches

Postby jaybea » Sun Jul 12, 2015 5:05 am

Do you have all the OS updates applied? Did you install ntpd or was it installed from the Network Time applicaiton.
I think that I have found the problem. I had the Network Time application installed, but it restarted at the end of January following an update and appears to have been running without a pid file since. Monit, which checks for the presence of the pid file, therefore thought it was not running and once every minute, restarted it. I have now reinstalled the Network Time app, and it is now running with a pid file and monit is happy again.

User avatar
bigfoot65
Project Manager
Posts: 10941
Joined: Mon May 25, 2009 4:31 pm

Re: Crashplan and inotify watches

Postby bigfoot65 » Sun Jul 12, 2015 7:29 am

Great to hear. Will consider this one resolved.
ßîgƒσστ65
Applications Manager

Who is online

Users browsing this forum: No registered users and 1 guest