SOLVED: Root full, so I moved the landing zone. Root is still full.

johnny5shared
Posts: 38
Joined: Sat Apr 02, 2016 1:48 am

SOLVED: Root full, so I moved the landing zone. Root is still full.

Postby johnny5shared » Tue Jul 05, 2016 8:55 pm

I have Greyhole working. I start moving files over the network and watching the GH logs, everything seemed great.

I quickly filled up my two (500GB) dedicated /gh drives, so I add another 1TB drive, and set it up with Greyhole (stop GH, run disk wizard, add disk in greyhole config), and kick off greyhole --fsck. The log indicates files are being moved around, according to rules in the GH config, over to my new drive... great!

Except that my root Amahi/Landing zone quickly fills up. It's only got 50GB spare and I have some large files and a lot of photos. Fair enough.

To get Amahi up I clear my landing zone according to the troubleshooter (and hope GH will repopulate the files... no biggie loosing the landing zone files since I was only redistributing files to my new drive) - I used...

Code: Select all

rm -rf /var/hda/files/shareName/*
...for each of my shares. Amahi comes back online from being locked up.

I follow the wiki to move the landing zone and run...

Code: Select all

mv /var/hda/files/videos /var/hda/files/drives/drive1/videos
...for each of my shares, then update the path in Amahi. I then run greyhole --fsck and away we go.

As far as I can tell, GH is configured correctly, to use my two original GH drives, and my new 1TB drive, which (for now) will also double as my landing zone.

However the root drive is still full. I can't for the life of me figure out how to clear it to allow GH to do it's thing.

sudo greyhole --empty-trash

produces...

PHP Fatal error: Uncaught exception 'Exception' with message 'SQLSTATE[HY000]: General error: 1021 Disk full (/var/tmp/#sql_3e1_0.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")' in /usr/bin/greyhole:599

I'm a bit stuck, but at least my files are on the GH drives still. I need help to...

1. Get root cleared of unnecessary files
2. Confirm the landing zone is correctly configured on my 1TB (Almost certain it is as I dropped a file in over the network, with GH disabled, and it ended up on the non boot drive, in the correct landing zone.)
3. Successfully run greyhole --fsck to get everything where it needs to be - currently nothing shows in my HDA, because I cleared out the old landing zone.
Last edited by johnny5shared on Wed Jul 06, 2016 7:09 pm, edited 1 time in total.

johnny5shared
Posts: 38
Joined: Sat Apr 02, 2016 1:48 am

Re: Root full, so I moved the landing zone. Root is still full.

Postby johnny5shared » Wed Jul 06, 2016 3:40 am

So I feel like I have my new drive added correctly now. I just have some crud somewhere in my boot drive.

I cleared out all the old mounts. I can't understand what's using up space in my root. It shouldn't be an issue now all my landing zones are on my separate 1TB drive.

Where can I look? Is there a cache or tmp that has blown out?

User avatar
bigfoot65
Project Manager
Posts: 11924
Joined: Mon May 25, 2009 4:31 pm

Re: Root full, so I moved the landing zone. Root is still full.

Postby bigfoot65 » Wed Jul 06, 2016 6:58 am

Have you tried a reboot?

Also might want to check /var/hda/files to see if there are files there or in /tmp that might be causing the issue.
ßîgƒσστ65
Applications Manager

My HDA: Intel(R) Core(TM) i5-3570K CPU @ 3.40GHz on MSI board, 16GB RAM, 1TBx1+2TBx2+4TBx2

johnny5shared
Posts: 38
Joined: Sat Apr 02, 2016 1:48 am

Re: Root full, so I moved the landing zone. Root is still full.

Postby johnny5shared » Wed Jul 06, 2016 4:32 pm

I have rebooted many times :) Always worthwhile trouble shooting advice though.

/var/hda/files only has 'drives'... this is normal yeah (if my landing zones are elsewhere)? This is only really a (sym) link to the other physical drives - yes? 'sudo du /var/hda/files -sh' Says 671GB - This seems to be the total of my two dedicated GH drives (500GB each). My root is only a 60GB drive. There's nothing else (e.g. old landing zones) in my root drive.

/tmp only has 32K
/var/hda/tmp only has 25M

I'm so close. But I can't run fsck because the drive is so full the database can't handle the queue.
Last edited by johnny5shared on Wed Jul 06, 2016 5:41 pm, edited 1 time in total.

johnny5shared
Posts: 38
Joined: Sat Apr 02, 2016 1:48 am

Re: Root full, so I moved the landing zone. Root is still full.

Postby johnny5shared » Wed Jul 06, 2016 4:58 pm

This may help (Please ignore the fact I called my 1TB drive 1gbDisk)

Code: Select all

Device Boot Start End Sectors Size Id Type /dev/sdd1 2048 976773119 976771072 465.8G 83 Linux [johnny5@localhost tmp]$ df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 1.5G 0 1.5G 0% /dev tmpfs 1.5G 0 1.5G 0% /dev/shm tmpfs 1.5G 65M 1.5G 5% /run tmpfs 1.5G 0 1.5G 0% /sys/fs/cgroup /dev/sda3 50G 50G 20K 100% / tmpfs 1.5G 32K 1.5G 1% /tmp /dev/sdc1 917G 73M 871G 1% /var/hda/files/drives/1gbDisk /dev/sdd1 459G 336G 100G 78% /var/hda/files/drives/500green /dev/sdb1 459G 335G 101G 77% /var/hda/files/drives/500blue /dev/sda1 477M 74M 374M 17% /boot none 4.0M 0 4.0M 0% /var/spool/greyhole/mem tmpfs 301M 0 301M 0% /run/user/1000
/dev/sda3 mounted at / is full. I've poked around looking at all the dir file sizes and there's nothing even remotely large I can see other than my /var/hda/files/drives. What is going on?

Code: Select all

$ sudo du /var/hda -h --max-depth=4 | sort -hr 671G /var/hda/files/drives 671G /var/hda/files 671G /var/hda 336G /var/hda/files/drives/500green/gh 336G /var/hda/files/drives/500green 335G /var/hda/files/drives/500blue/gh 335G /var/hda/files/drives/500blue 97M /var/hda/web-apps/owncloud8/html 97M /var/hda/web-apps/owncloud8 97M /var/hda/web-apps 77M /var/hda/platform 76M /var/hda/platform/html 65M /var/hda/platform/html/vendor/bundle 65M /var/hda/platform/html/vendor 37M /var/hda/web-apps/owncloud8/html/apps 27M /var/hda/web-apps/owncloud8/html/core 26M /var/hda/web-apps/owncloud8/html/3rdparty 25M /var/hda/tmp/amahi-download-cache 25M /var/hda/tmp 5.6M /var/hda/web-apps/owncloud8/html/lib 4.3M /var/hda/platform/html/tmp/cache 4.3M /var/hda/platform/html/tmp 2.8M /var/hda/web-apps/owncloud8/html/settings 2.3M /var/hda/platform/html/plugins 1.6M /var/hda/platform/logs 1.6M /var/hda/platform/html/plugins/1001-disk_wizards 1.6M /var/hda/platform/html/log 1.5M /var/hda/platform/html/app 944K /var/hda/platform/html/app/assets 916K /var/hda/platform/html/config 880K /var/hda/files/drives/1gbDisk 836K /var/hda/files/drives/1gbDisk/MarkGH 820K /var/hda/platform/html/config/locales 380K /var/hda/platform/html/app/views 296K /var/hda/web-apps/owncloud8/html/config 152K /var/hda/platform/html/plugins/040-apps 144K /var/hda/apps/520ut3lo6w/elevated 144K /var/hda/apps/520ut3lo6w 144K /var/hda/apps 136K /var/hda/platform/html/plugins/050-network 128K /var/hda/platform/html/lib 116K /var/hda/platform/html/plugins/020-shares 116K /var/hda/platform/html/db 112K /var/hda/platform/html/bin 104K /var/hda/platform/html/public 104K /var/hda/platform/html/plugins/010-users 100K /var/hda/platform/html/app/models 92K /var/hda/platform/html/plugins/080-settings 80K /var/hda/platform/html/plugins/030-disks 76K /var/hda/platform/html/db/migrate 72K /var/hda/platform/html/app/controllers 64K /var/hda/platform/html/spec 52K /var/hda/web-apps/owncloud8/html/themes 48K /var/hda/platform/html/lib/generators 40K /var/hda/platform/html/config/initializers 36K /var/hda/platform/html/spec/features 32K /var/hda/platform/html/test 28K /var/hda/platform/html/db/sample-data 28K /var/hda/platform/html/app/helpers 24K /var/hda/web-apps/owncloud8/html/scripts 24K /var/hda/platform/html/script 16K /var/hda/web-apps/owncloud8/logs 16K /var/hda/platform/html/config/environments 16K /var/hda/files/drives/500green/lost+found 16K /var/hda/files/drives/500blue/lost+found 12K /var/hda/web-apps/owncloud8/html/ocs 12K /var/hda/platform/html/test/unit 12K /var/hda/platform/html/test/functional 12K /var/hda/platform/html/lib/tasks 8.0K /var/hda/web-apps/owncloud8/html/ocs-provider 8.0K /var/hda/web-apps/owncloud8/html/l10n 8.0K /var/hda/platform/html/spec/models 8.0K /var/hda/files/drives/1gbDisk/gh 4.0K /var/hda/platform/html/test/performance 4.0K /var/hda/platform/html/spec/support 4.0K /var/hda/platform/html/misc 4.0K /var/hda/platform/html/lib/templates 4.0K /var/hda/platform/html/.bundle 4.0K /var/hda/platform/html/app/validators 4.0K /var/hda/files/drives/1gbDisk/Videos 4.0K /var/hda/files/drives/1gbDisk/TV 4.0K /var/hda/files/drives/1gbDisk/Software 4.0K /var/hda/files/drives/1gbDisk/Public 4.0K /var/hda/files/drives/1gbDisk/Pictures 4.0K /var/hda/files/drives/1gbDisk/Owncloud8 4.0K /var/hda/files/drives/1gbDisk/Music 4.0K /var/hda/files/drives/1gbDisk/Movies 4.0K /var/hda/domain-settings/netlogon 4.0K /var/hda/domain-settings 0 /var/hda/shares ...

johnny5shared
Posts: 38
Joined: Sat Apr 02, 2016 1:48 am

Re: Root full, so I moved the landing zone. Root is still full.

Postby johnny5shared » Wed Jul 06, 2016 7:49 pm

From the troubleshooter...
Blow away the contents of your landing zone shares. It is very important you don't touch the /drives folder within /var/hda/files if you have a default setup.
What does this mean exactly? Do I have a default setup now? What purpose does this dir serve with an external landing zone? I assumed I couldn't blow that away because it's my only copy of my files. But it seems this dir is all that's taking up space on my root. Is it just a symlink/alias/mount to external physical drives? Or is there crud in this location?

johnny5shared
Posts: 38
Joined: Sat Apr 02, 2016 1:48 am

Re: Root full, so I moved the landing zone. Root is still full.

Postby johnny5shared » Thu Jul 07, 2016 5:06 am

I was able to clean out a few hundred MB by running...

Code: Select all

dnf clean all --enablerepo=\*
I cleared the GH queue and miraculously was able to run a whole greyhole --fsck, with 1.6MB to spare! I have to admit it got a bit hairy when I realised some of my mac .mpkg files were being broken down to individual contents files, which were filling up my database and GH on my boot.

Anyway. Greyhole is back, and using the external landing zone.

But I still have a full boot /

As soon as I start using my GH shares, the database on root will quickly fill the remaining 1.6MB

:x
Last edited by johnny5shared on Thu Jul 07, 2016 2:50 pm, edited 1 time in total.

User avatar
bigfoot65
Project Manager
Posts: 11924
Joined: Mon May 25, 2009 4:31 pm

Re: Root full, so I moved the landing zone. Root is still full.

Postby bigfoot65 » Thu Jul 07, 2016 2:17 pm

Try executing this command as root user:

Code: Select all

du -a /var | sort -n -r | head -n 10
It should tell you the top 10 directories with the most space used.

Then use that to see where your / drive space has gone. Could be hidden files.
ßîgƒσστ65
Applications Manager

My HDA: Intel(R) Core(TM) i5-3570K CPU @ 3.40GHz on MSI board, 16GB RAM, 1TBx1+2TBx2+4TBx2

johnny5shared
Posts: 38
Joined: Sat Apr 02, 2016 1:48 am

Re: Root full, so I moved the landing zone. Root is still full.

Postby johnny5shared » Thu Jul 07, 2016 5:15 pm

That just lists GH dir's. These are virtually mounted into root yeah? I asked about this on a previous post.

Code: Select all

302958768 /var/hda/files/drives/500blue/gh 300926956 /var/hda/files/drives/500green 300926936 /var/hda/files/drives/500green/gh 161112132 /var/hda/files/drives/500blue/gh/Pictures 161111908 /var/hda/files/drives/500green/gh/Pictures

User avatar
bigfoot65
Project Manager
Posts: 11924
Joined: Mon May 25, 2009 4:31 pm

Re: Root full, so I moved the landing zone. Root is still full.

Postby bigfoot65 » Thu Jul 07, 2016 5:24 pm

That just lists GH dir's. These are virtually mounted into root yeah? I asked about this on a previous post.
That is correct, but there is something else on the root drive taking up space.

Might try unmounting all the GH drives, then run the script again.

Also what does your /etc/fstab look like?

I presume your root drive is not in the Greyhole Storage Pool, correct?
ßîgƒσστ65
Applications Manager

My HDA: Intel(R) Core(TM) i5-3570K CPU @ 3.40GHz on MSI board, 16GB RAM, 1TBx1+2TBx2+4TBx2

Who is online

Users browsing this forum: No registered users and 25 guests