SOLVED: Root full after botched GOING command

evylrat
Posts: 106
Joined: Sun Jan 02, 2011 6:37 pm

SOLVED: Root full after botched GOING command

Postby evylrat » Fri Sep 01, 2017 8:07 am

The issue: A drive of mine is failing; SMART is reporting temperatures that are not accurate (over 100 degrees) - so I've got to RMA it.
I've got my last HDA box's 2TB still sitting around, so I'll use one of them to replace the failing drive.
I used Disk Wizard to add the new drive I'd inserted and it appeared to add it and format it. I saw that it was mounted ok.
I used Greyhole UI to add the drive to the pool. I unchecked my bad drive.
Check the greyhole log, nothing was happening so I used the greyhole --going command to remove the drive. It began trawling through the storage. But shortly after, my HDA went offline; no shares and dashboard showed the DEBUG page.
I checked greyhole from the command line and it did NOT show the drive I'd added to the pool. So I guess it's tried to create copies of shares in my landing zone.
I got the new drive added to the pool the manual way and checked that it was working. I'd also left the drive removed with the going command in, and ran an FSCK to try and repopulate my shares fully.
I ran a balance command, the new drive is filling up, but my root is still full (only a 9% free now).
I cleared the greyhole work queue and reran FSCK to see if it would move files from the LZ back to storage, it didn't.
I noticed that all the files on the drive I tried to remove also went into "greyhole trash".

APASTE (0.3.8.1) isn't working, says "Error: Server did not return a correct JSON response" (*fixed!)
but here you go, https://paste.fedoraproject.org/paste/x ... bWVAaKXtJw
I've not run the --going command again till I free up my space on the OS drive, which is usually nearly empty.
Log extracts -
httpd/error_log - https://paste.fedoraproject.org/paste/j ... Jh07tH5LHg
mariadb.log - https://paste.fedoraproject.org/paste/m ... xig3bQXzIA
dnf.log - https://paste.fedoraproject.org/paste/q ... phVhz~jBAQ
Lots of critcal errors in the dnf.log, I removed the lock file, updated FPASTE and tried APASTE again - now it works.

The files I suspect are in /var/hda/files/drives
  • /var/hda/
    4.0K ./domain-settings
    239M ./platform
    152K ./apps
    172K ./dbs
    0 ./drives
    0 ./elevated
    0 ./shares
    21M ./tmp
    1.2G ./web-apps
    5.8T ./files
Short version: I ran the --going command without enough free space in the pool, now I think there are files in the LZ, but how do I distinguish symblinks from actual files? I'm thinking I could move them out of the pool, then back in so greyhole fixes it.

Thank you in advance for your time looking at this.

User avatar
bigfoot65
Project Manager
Posts: 11924
Joined: Mon May 25, 2009 4:31 pm

Re: Root full after botched GOING command

Postby bigfoot65 » Tue Sep 05, 2017 6:02 am

how do I distinguish symblinks from actual files?
From command line, you can do:

Code: Select all

ls -al /var/hda/files/sharename
You will easily be able to see what files are symbolic linked.
ßîgƒσστ65
Applications Manager

My HDA: Intel(R) Core(TM) i5-3570K CPU @ 3.40GHz on MSI board, 16GB RAM, 1TBx1+2TBx2+4TBx2

evylrat
Posts: 106
Joined: Sun Jan 02, 2011 6:37 pm

Re: Root full after botched GOING command

Postby evylrat » Tue Sep 05, 2017 3:04 pm

Finally I find one of your old posts, viewtopic.php?f=39&t=6538&start=10#p37130
This thread has been useful. Sounds like what may have happened to my setup.
So I've unmounted my 4 nas drives; 2 existing (nas1 and nas2), 1 failing (nas3) and the recently added drive to move files to (nas4).
I ran

Code: Select all

du -sh /var/hda/files/drives/*
  • 4.0K nas1
    4.0K nas2
    7.3G nas3
    179G nas4
So... move these files from /var/hda/files/drives/ ? Get greyhole running ok then copy them back if they don't already exist?

evylrat
Posts: 106
Joined: Sun Jan 02, 2011 6:37 pm

Re: Root full after botched GOING command

Postby evylrat » Thu Sep 14, 2017 12:05 am

Feel free to mark as solved!

User avatar
bigfoot65
Project Manager
Posts: 11924
Joined: Mon May 25, 2009 4:31 pm

Re: Root full after botched GOING command

Postby bigfoot65 » Thu Sep 14, 2017 2:41 am

Glad to hear you got it sorted out.

Marking as closed.
ßîgƒσστ65
Applications Manager

My HDA: Intel(R) Core(TM) i5-3570K CPU @ 3.40GHz on MSI board, 16GB RAM, 1TBx1+2TBx2+4TBx2

Who is online

Users browsing this forum: No registered users and 6 guests