Greyhole Drive Not Balancing and also Full

heelphan
Posts: 3
Joined: Tue Dec 23, 2014 9:01 am

Greyhole Drive Not Balancing and also Full

Postby heelphan » Tue Dec 23, 2014 9:42 am

Some backgrund (I apologize if this gets long winded, just want to be thorough):
I have been using Amahi for about 4 years now, coming from WHS. I still consider myself a noob when it comes to Linux, even though I've been using this system for 4 years. After being on Amahi 6 and Fedora 14 the entire time, I decided to finally upgrde to Amahi 7 and Fedora 19 this past summer. At the same time, I upgraded my OS drive to a 128GB SSD. I have 5 storage drives that I transferred from Amahi 6, a 1TB, (2) 2TB,a 1.5TB and a 3 TB. One of these storage drives was used as my OS drive in my old config. I didn't do anything to erase these files on that drive (and really didn't know how to).

My problem started recetnly when I tried to copy a movie into my Videos share. It was a 6GB movie and it told me there was not enough space to copy the movie. I checked greyhole --stats and I noticed one of my drives only had maybe 3GB free. So, I thought I needed to do a balance. I balanced and that particaular drive (/var/hda/files/gh when I do greyhole --stats) would not balance. The other drives were balancing with each other but it was like that drive was not being seen (this is one of the 2TB drives, and possibly the drive that Amahi had been installed on as the OS drive in the previous system).

I then read that it is not advised to have /var/hda/files as one of my storage pool drives, so I unchecked it on the drive pooling dashboard. That freed up some space on it, but I still was unable to copy many files to the Videos share before it said it was full again. I then tried to move the landing zone for Videos to my 3TB drive since it was the largest and had the most space free. Unfortunaltey, it was taking way too long to move the /var/hda/files/videos folder, so I stopped it. It was also filling up the drive rapdily.

What can I do to fix this situation? I need the pool to recognize that 2TB drive again to balance it and remove what I'm sure are some duplicate files that I don't need now. (I only duplicate my Photos and Home Movies folders).

Here's my info, any help would be greatly appreciated:

Code: Select all

3.14.17-100.fc19.x86_64 samba-4.0.21-1.fc19.x86_64 amahi-greyhole-0.9.60-2.x86_64
smb.conf: http://ur1.ca/j7i6i
greyhole.conf: http://ur1.ca/j7i7r

Code: Select all

[root@localhost heelphan]# mount proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,size=2000664k,nr_inodes=500166,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) /dev/sda6 on / type ext4 (rw,relatime,data=ordered) systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=38,pgrp=1,timeout=300,minproto=5,maxproto=5,direct) mqueue on /dev/mqueue type mqueue (rw,relatime) tmpfs on /tmp type tmpfs (rw) configfs on /sys/kernel/config type configfs (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) sunrpc on /proc/fs/nfsd type nfsd (rw,relatime) /dev/sda2 on /home type ext4 (rw,relatime,data=ordered) /dev/sda1 on /boot type ext4 (rw,relatime,data=ordered) /dev/sdd3 on /var/hda/files type ext4 (rw,relatime,data=ordered) /dev/sdc1 on /var/hda/files/drives/drive7 type ext4 (rw,relatime,data=ordered) /dev/sde1 on /var/hda/files/drives/drive9 type ext4 (rw,relatime,data=ordered) /dev/sdf1 on /var/hda/files/drives/drive2 type ext4 (rw,relatime,data=ordered) /dev/sdb1 on /var/hda/files/drives/drive8 type ext4 (rw,relatime,data=ordered) none on /var/spool/greyhole/mem type tmpfs (rw,relatime,size=4096k) fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime) /dev/sda5 on /run/media/heelphan/986f64c8-b879-4e5c-a89f-0d82d9d7e4b3 type ext4 (rw,nosuid,nodev,relatime,data=ordered,uhelper=udisks2) /dev/sdd1 on /run/media/heelphan/02e73f16-b02a-478b-a0df-5fb234c97d83 type ext4 (rw,nosuid,nodev,relatime,data=ordered,uhelper=udisks2) gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=100) //127.0.0.1/Home Movies on /mnt/samba/Home Movies type cifs (rw,relatime,vers=1.0,cache=strict,username=heelphan,domain=HOME,uid=1000,forceuid,gid=100,forcegid,addr=127.0.0.1,file_mode=0660,dir_mode=0770,hard,nounix,nobrl,mfsymlinks,rsize=61440,wsize=65536,actimeo=1) //127.0.0.1/Macbook on /mnt/samba/Macbook type cifs (rw,relatime,vers=1.0,cache=strict,username=heelphan,domain=HOME,uid=1000,forceuid,gid=100,forcegid,addr=127.0.0.1,file_mode=0660,dir_mode=0770,hard,nounix,nobrl,mfsymlinks,rsize=61440,wsize=65536,actimeo=1) //127.0.0.1/Music on /mnt/samba/Music type cifs (rw,relatime,vers=1.0,cache=strict,username=heelphan,domain=HOME,uid=1000,forceuid,gid=100,forcegid,addr=127.0.0.1,file_mode=0660,dir_mode=0770,hard,nounix,nobrl,mfsymlinks,rsize=61440,wsize=65536,actimeo=1) //127.0.0.1/Owncloud on /mnt/samba/Owncloud type cifs (rw,relatime,vers=1.0,cache=strict,username=heelphan,domain=HOME,uid=1000,forceuid,gid=100,forcegid,addr=127.0.0.1,file_mode=0660,dir_mode=0770,hard,nounix,nobrl,mfsymlinks,rsize=61440,wsize=65536,actimeo=1) //127.0.0.1/Photos on /mnt/samba/Photos type cifs (rw,relatime,vers=1.0,cache=strict,username=heelphan,domain=HOME,uid=1000,forceuid,gid=100,forcegid,addr=127.0.0.1,file_mode=0660,dir_mode=0770,hard,nounix,nobrl,mfsymlinks,rsize=61440,wsize=65536,actimeo=1) //127.0.0.1/Users on /mnt/samba/Users type cifs (rw,relatime,vers=1.0,cache=strict,username=heelphan,domain=HOME,uid=1000,forceuid,gid=100,forcegid,addr=127.0.0.1,file_mode=0660,dir_mode=0770,hard,nounix,nobrl,mfsymlinks,rsize=61440,wsize=65536,actimeo=1) //127.0.0.1/Videos on /mnt/samba/Videos type cifs (rw,relatime,vers=1.0,cache=strict,username=heelphan,domain=HOME,uid=1000,forceuid,gid=100,forcegid,addr=127.0.0.1,file_mode=0660,dir_mode=0770,hard,nounix,nobrl,mfsymlinks,rsize=61440,wsize=65536,actimeo=1)

Code: Select all

[root@localhost heelphan]# fdisk -l Disk /dev/sda: 128.0 GB, 128035676160 bytes, 250069680 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: dos Disk identifier: 0x000c9871 Device Boot Start End Blocks Id System /dev/sda1 * 2048 411647 204800 83 Linux /dev/sda2 411648 136435711 68012032 83 Linux /dev/sda3 136435712 144594943 4079616 82 Linux swap / Solaris /dev/sda4 144594944 250069679 52737368 5 Extended /dev/sda5 144596992 152756223 4079616 83 Linux /dev/sda6 152758272 250068991 48655360 83 Linux Disk /dev/sdb: 3000.6 GB, 3000592982016 bytes, 5860533168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disk label type: dos Disk identifier: 0x00000000 Device Boot Start End Blocks Id System /dev/sdb1 1 4294967295 2147483647+ ee GPT Partition 1 does not start on physical sector boundary. Disk /dev/sdc: 1000.2 GB, 1000204886016 bytes, 1953525168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: dos Disk identifier: 0x000c826e Device Boot Start End Blocks Id System /dev/sdc1 2048 1953523711 976760832 83 Linux Disk /dev/sdd: 2000.4 GB, 2000398934016 bytes, 3907029168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: dos Disk identifier: 0x0ab29f3e Device Boot Start End Blocks Id System /dev/sdd1 * 2048 1026047 512000 83 Linux /dev/sdd2 1026048 70658047 34816000 8e Linux LVM /dev/sdd3 70658048 3907028991 1918185472 83 Linux Disk /dev/sde: 1500.3 GB, 1500301910016 bytes, 2930277168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: dos Disk identifier: 0x00000000 Device Boot Start End Blocks Id System /dev/sde1 * 63 2930277167 1465138552+ 83 Linux Disk /dev/sdf: 2000.4 GB, 2000398934016 bytes, 3907029168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: dos Disk identifier: 0x00089dd5 Device Boot Start End Blocks Id System /dev/sdf1 1 3907029167 1953514583+ 83 Linux Disk /dev/mapper/vg_homeserver-lv_swap: 4194 MB, 4194304000 bytes, 8192000 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/mapper/vg_homeserver-lv_root: 20.9 GB, 20937965568 bytes, 40894464 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/mapper/vg_homeserver-lv_home: 10.5 GB, 10502537216 bytes, 20512768 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes

Code: Select all

[root@localhost heelphan]# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda6 46G 11G 34G 24% / devtmpfs 2.0G 0 2.0G 0% /dev tmpfs 2.0G 116K 2.0G 1% /dev/shm tmpfs 2.0G 732K 2.0G 1% /run tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup tmpfs 2.0G 148K 2.0G 1% /tmp /dev/sda2 64G 327M 61G 1% /home /dev/sda1 190M 77M 100M 44% /boot /dev/sdd3 1.8T 1.8T 383M 100% /var/hda/files /dev/sdc1 917G 692G 180G 80% /var/hda/files/drives/drive7 /dev/sde1 1.4T 1.1T 194G 86% /var/hda/files/drives/drive9 /dev/sdf1 1.8T 1.6T 179G 90% /var/hda/files/drives/drive2 /dev/sdb1 2.8T 2.5T 138G 95% /var/hda/files/drives/drive8 none 4.0M 0 4.0M 0% /var/spool/greyhole/mem /dev/sda5 3.8G 7.8M 3.6G 1% /run/media/heelphan/986f64c8-b879-4e5c-a89f-0d82d9d7e4b3 /dev/sdd1 477M 21M 427M 5% /run/media/heelphan/02e73f16-b02a-478b-a0df-5fb234c97d83 //127.0.0.1/Home Movies 8.6T 7.9T 690G 93% /mnt/samba/Home Movies //127.0.0.1/Macbook 1.8T 1.8T 383M 100% /mnt/samba/Macbook //127.0.0.1/Music 1.8T 1.8T 383M 100% /mnt/samba/Music //127.0.0.1/Owncloud 1.8T 1.8T 383M 100% /mnt/samba/Owncloud //127.0.0.1/Photos 8.6T 7.9T 690G 93% /mnt/samba/Photos //127.0.0.1/Users 1.8T 1.8T 383M 100% /mnt/samba/Users //127.0.0.1/Videos 2.8T 2.6T 138G 96% /mnt/samba/Videos

Code: Select all

[root@localhost heelphan]# greyhole --stats Greyhole Statistics =================== Storage Pool Total - Used = Free + Trash = Possible /var/hda/files/drives/drive7/gh: 917G - 691G = 179G + 0G = 179G /var/hda/files/drives/drive2/gh: 1834G - 1562G = 178G + 0G = 178G /var/hda/files/drives/drive9/gh: 1375G - 1112G = 194G + 0G = 194G /var/hda/files/drives/drive8/gh: 2794G - 2516G = 138G + 0G = 138G /var/hda/files/gh: 1800G - 1764G = 0G + 0G = 0G ========================================== Total: 8720G - 7645G = 689G + 0G = 689G

Code: Select all

[root@localhost heelphan]# mysql -u root -phda -e "select * from disk_pool_partitions" hda_production +----+------------------------------+--------------+---------------------+---------------------+ | id | path | minimum_free | created_at | updated_at | +----+------------------------------+--------------+---------------------+---------------------+ | 2 | /var/hda/files/drives/drive7 | 10 | 2014-09-18 14:24:13 | 2014-09-18 14:24:13 | | 3 | /var/hda/files/drives/drive2 | 10 | 2014-09-18 14:24:16 | 2014-09-18 14:24:16 | | 4 | /var/hda/files/drives/drive9 | 10 | 2014-09-18 14:24:17 | 2014-09-18 14:24:17 | | 5 | /var/hda/files/drives/drive8 | 10 | 2014-09-18 14:24:18 | 2014-09-18 14:24:18 | | 6 | /var/hda/files | 10 | 2014-12-22 03:53:58 | 2014-12-22 03:53:58 | +----+------------------------------+--------------+---------------------+---------------------+
Directories on the root drives: http://ur1.ca/j7iax

Code: Select all

[root@localhost heelphan]# greyhole --view-queue Greyhole Work Queue Statistics ============================== This table gives you the number of pending operations queued for the Greyhole daemon, per share. Write Delete Rename Check Home Movies 0 0 0 0 Photos 0 0 0 0 =============================================== Total 0 0 0 0 Write Delete Rename Check The following is the number of pending operations that the Greyhole daemon still needs to parse. Until it does, the nature of those operations is unknown. Spooled operations that have been parsed will be listed above and disappear from the count below. Spooled 0
Thanks for any help you all can provide!

User avatar
bigfoot65
Project Manager
Posts: 11924
Joined: Mon May 25, 2009 4:31 pm

Re: Greyhole Drive Not Balancing and also Full

Postby bigfoot65 » Tue Dec 23, 2014 2:18 pm

Did you follow the wiki guidance for moving the LZ?

Recommend you stop Greyhole temporarily until you can get this sorted out. It will restart automatically if you don't stop it via the Dashboard and disable the watchdog service.

Once you have things in a state, recommend you determine what files (if any) can be removed to free up space on the LZ drive.

As you have stated, not a good idea to have /var/hda/files as part of the storage pool. This is likely what caused the problems. Now to get it fixed may take some patience and time.

Will try to help as much as possible.
ßîgƒσστ65
Applications Manager

My HDA: Intel(R) Core(TM) i5-3570K CPU @ 3.40GHz on MSI board, 16GB RAM, 1TBx1+2TBx2+4TBx2

heelphan
Posts: 3
Joined: Tue Dec 23, 2014 9:01 am

Re: Greyhole Drive Not Balancing and also Full

Postby heelphan » Tue Dec 23, 2014 8:56 pm

Thanks BigFoot.

I have stopped the Greyhole service and the watchdog.

Where should I start looking for files that can be deleted? Where is it safe to delete files? I'm a little confused by that. Should I delete files from /var/hda/files/videos or /var/hda/files/gh? I know that there are symbolic links to files and then there are the actual files. Where are these each stored and which should I be deleting?

Is there a Linux program that's good at finding duplicate files for you that would make it easier on me?

Thanks!

User avatar
bigfoot65
Project Manager
Posts: 11924
Joined: Mon May 25, 2009 4:31 pm

Re: Greyhole Drive Not Balancing and also Full

Postby bigfoot65 » Wed Dec 24, 2014 11:48 am

Ok this is a area that you need to be careful. How many copies of files did you have Greyhole set to do? Do you have a backup of the files in question?

Ideally you would want to remove the /var/hda/files/gh directory. That is where the extra file copies are located that have caused the issue. Recommend you only do this if there is a copy of the files on another drive.

You could move the directory to another drive temporarily to get things going again. If you do so, ensure you don't overwrite the gh directory on that drive.

Once you remove this directory, you can enable Greyhole and force a FSCK.
ßîgƒσστ65
Applications Manager

My HDA: Intel(R) Core(TM) i5-3570K CPU @ 3.40GHz on MSI board, 16GB RAM, 1TBx1+2TBx2+4TBx2

heelphan
Posts: 3
Joined: Tue Dec 23, 2014 9:01 am

Re: Greyhole Drive Not Balancing and also Full

Postby heelphan » Wed Dec 24, 2014 2:14 pm

Well, the ones that are taking up the most space are in the Videos share. I do not have a backup of those files and I did not have Greyhole add an extra copy of that share.

What's the easiest way to find duplicates, just manually look in each of the GH directories for the same files that are on the /var/hda/files/gh directory and if I find a duplicate delete it from /var/hda/files/gh and if not, copy it to the GH folder on another drive with enough space? I might not be able to accomplish this unless a I buy another 2TB hard drive!

Thanks for all of your help!

User avatar
bigfoot65
Project Manager
Posts: 11924
Joined: Mon May 25, 2009 4:31 pm

Re: Greyhole Drive Not Balancing and also Full

Postby bigfoot65 » Wed Dec 24, 2014 2:29 pm

It's best not to mess with files inside the gh folders. I would recommend you copy the files from the Video share on /var/hda/files/gh/Videos to a safe place. Then remove the directory as indicated in my last post.

You could use one of the other drives to temporarily hold those files until you get this sorted out. If you have not specified additional copies of any shares, then there are no duplicates.

You may want to read the wiki guidance regarding Greyhole to better understand how it works if you have not already done so. It's important to understand how it works so you don't make your situation any worse.
ßîgƒσστ65
Applications Manager

My HDA: Intel(R) Core(TM) i5-3570K CPU @ 3.40GHz on MSI board, 16GB RAM, 1TBx1+2TBx2+4TBx2

Who is online

Users browsing this forum: No registered users and 9 guests