SOLVED: root filesystem not writeable

[ghostwheel]

HDA has been working great for several months. Came home today to find out the file server was no longer working and the dashboard wouldn't come up. Webmin is partially working, but many functions are complaining that there is No space left on device. For example, the Apache Webserver link shows: Failed to open /etc/webmin/apache/site for writing : No space left on device

Rebooted, no change.

SSH'd into server and this is what I get for a df -h (cutting off the user shares which all show a lot of space)
[root@localhost /]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/fedora-root 50G 33G 14G 71% /
devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 1.9G 588K 1.9G 1% /run
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
tmpfs 1.9G 236K 1.9G 1% /tmp
/dev/sda1 477M 62M 390M 14% /boot
/dev/mapper/fedora-home 57G 49G 4.6G 92% /home
/dev/sdb1 3.6T 2.1T 1.4T 62% /var/hda/drives/drive1
/dev/sdc1 3.6T 2.1T 1.4T 62% /var/hda/drives/drive2
none 4.0M 0 4.0M 0% /var/spool/greyhole/mem

I tried creating a file under /, and get "No space left on device".

I'm not sure what to try next. Suggestions?

[bigfoot65]

You need to clear some space on the / partition.

Do you have it included as part of the Greyhole storage pool? Is that your landing zone?

This is often the case when the drive becomes full.

Has Greyhole stopped?

[ghostwheel]

greyhole has stopped. Landing zone is under /home which is a different partition and not full. I freed up some space on / by deleting some logs and it is showing 21G available (56% used), so it should have plenty of space, but it continues to report no space available.

[bigfoot65]

Do you have all the latest OS updates?

Code: Select all

yum update

Next would be to check log files, i.e. /var/log/greyhole.log and possibly /var/log/messages.
If you post any of the details here, please attach vs posting a lot of lines of text.

[ghostwheel]

Turns out the inode table was at 100% (df -i). Cleared the greyhole spool folder and the inode table went to 4% and everything came back to life

So now my questions are, is there a reason is filled up and is there anything I can do to prevent this in the future?

[bigfoot65]

Glad to hear you sorted it out.

Not sure what could be the cause or if it would return. Recommend you file a bug report with the Greyhole author. I believe the site is https://github.com/gboudreau/Greyhole/issues.

This is isolated to Greyhole most likely, so best to go directly to the source.

EDIT: This might help https://github.com/gboudreau/Greyhole/issues/123