Page 1 of 1

SOLVED: Root full after botched GOING command

Posted: Fri Sep 01, 2017 8:07 am
by evylrat
The issue: A drive of mine is failing; SMART is reporting temperatures that are not accurate (over 100 degrees) - so I've got to RMA it.
I've got my last HDA box's 2TB still sitting around, so I'll use one of them to replace the failing drive.
I used Disk Wizard to add the new drive I'd inserted and it appeared to add it and format it. I saw that it was mounted ok.
I used Greyhole UI to add the drive to the pool. I unchecked my bad drive.
Check the greyhole log, nothing was happening so I used the greyhole --going command to remove the drive. It began trawling through the storage. But shortly after, my HDA went offline; no shares and dashboard showed the DEBUG page.
I checked greyhole from the command line and it did NOT show the drive I'd added to the pool. So I guess it's tried to create copies of shares in my landing zone.
I got the new drive added to the pool the manual way and checked that it was working. I'd also left the drive removed with the going command in, and ran an FSCK to try and repopulate my shares fully.
I ran a balance command, the new drive is filling up, but my root is still full (only a 9% free now).
I cleared the greyhole work queue and reran FSCK to see if it would move files from the LZ back to storage, it didn't.
I noticed that all the files on the drive I tried to remove also went into "greyhole trash".

APASTE (0.3.8.1) isn't working, says "Error: Server did not return a correct JSON response" (*fixed!)
but here you go, https://paste.fedoraproject.org/paste/x ... bWVAaKXtJw
I've not run the --going command again till I free up my space on the OS drive, which is usually nearly empty.
Log extracts -
httpd/error_log - https://paste.fedoraproject.org/paste/j ... Jh07tH5LHg
mariadb.log - https://paste.fedoraproject.org/paste/m ... xig3bQXzIA
dnf.log - https://paste.fedoraproject.org/paste/q ... phVhz~jBAQ
Lots of critcal errors in the dnf.log, I removed the lock file, updated FPASTE and tried APASTE again - now it works.

The files I suspect are in /var/hda/files/drives
  • /var/hda/
    4.0K ./domain-settings
    239M ./platform
    152K ./apps
    172K ./dbs
    0 ./drives
    0 ./elevated
    0 ./shares
    21M ./tmp
    1.2G ./web-apps
    5.8T ./files
Short version: I ran the --going command without enough free space in the pool, now I think there are files in the LZ, but how do I distinguish symblinks from actual files? I'm thinking I could move them out of the pool, then back in so greyhole fixes it.

Thank you in advance for your time looking at this.

Re: Root full after botched GOING command

Posted: Tue Sep 05, 2017 6:02 am
by bigfoot65
how do I distinguish symblinks from actual files?
From command line, you can do:

Code: Select all

ls -al /var/hda/files/sharename
You will easily be able to see what files are symbolic linked.

Re: Root full after botched GOING command

Posted: Tue Sep 05, 2017 3:04 pm
by evylrat
Finally I find one of your old posts, viewtopic.php?f=39&t=6538&start=10#p37130
This thread has been useful. Sounds like what may have happened to my setup.
So I've unmounted my 4 nas drives; 2 existing (nas1 and nas2), 1 failing (nas3) and the recently added drive to move files to (nas4).
I ran

Code: Select all

du -sh /var/hda/files/drives/*
  • 4.0K nas1
    4.0K nas2
    7.3G nas3
    179G nas4
So... move these files from /var/hda/files/drives/ ? Get greyhole running ok then copy them back if they don't already exist?

Re: Root full after botched GOING command

Posted: Thu Sep 14, 2017 12:05 am
by evylrat
Feel free to mark as solved!

Re: Root full after botched GOING command

Posted: Thu Sep 14, 2017 2:41 am
by bigfoot65
Glad to hear you got it sorted out.

Marking as closed.