Files Missing from Directories - Queue Steadily Growing

[mattrountree]

Files have been disappearing from my Greyhole directory. I witnessed the files in one directory disappear in real time. I attempted the fixes (fsck, file permissions, switch to mysql) suggested in the many other posts with similar issue but haven't been able to resolve anything. I think the files are moving from the landing zone but symbolic links aren't being created. The other suspicious thing is the greyhole queue has been steadily rising for the last few weeks. I have run greyhole --fsck several times as well as reset the machine but the queue keeps increasing. It was my intention to resolve this issue before upgrading Amahi. Any advice on fixing these files is appreciated.

Code: Select all

uname -r; rpm -q samba hda-greyhole 2.6.32.26-175.fc12.x86_64 samba-3.4.9-60.fc12.x86_64 hda-greyhole-0.7.5-1.x86_64

Code: Select all

yum -y install fpaste; fpaste /etc/samba/smb.conf; fpaste /etc/greyhole.conf http://fpaste.org/Kgtq/ http://fpaste.org/dh6B/

Code: Select all

mount; fdisk -l; df -h; greyhole --stats /dev/mapper/vg_linuxserver-lv_root on / type ext4 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw) /dev/sda1 on /boot type ext4 (rw) /dev/sdf1 on /var/hda/files/drives/drive3 type ext4 (rw) /dev/sdd1 on /var/hda/files/drives/drive4 type ext4 (rw) /dev/sde1 on /var/hda/files/drives/drive5 type ext4 (rw) /dev/sdc1 on /var/hda/files/drives/drive6 type ext4 (rw) /dev/sdb1 on /var/hda/files/drives/drive7 type ext4 (rw) /dev/sdg1 on /var/hda/files/drives/drive9 type ext4 (rw) /dev/sdh1 on /var/hda/files/drives/drive11 type ext4 (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) funionfs on /var/hda/apps/ushare/files type fuse.funionfs (rw,nosuid,nodev,allow_other) gvfs-fuse-daemon on /home/matt/.gvfs type fuse.gvfs-fuse-daemon (rw,nosuid,nodev,user=matt) Disk /dev/sda: 2000.4 GB, 2000398934016 bytes 255 heads, 63 sectors/track, 243201 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x00098578 Device Boot Start End Blocks Id System /dev/sda1 * 1 26 204800 83 Linux Partition 1 does not end on cylinder boundary. /dev/sda2 26 243201 1953307201 8e Linux LVM Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes 255 heads, 63 sectors/track, 121601 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x6c6df128 Device Boot Start End Blocks Id System /dev/sdb1 1 121601 976760001 83 Linux Disk /dev/sdc: 2000.4 GB, 2000398934016 bytes 255 heads, 63 sectors/track, 243201 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x8f41220a Device Boot Start End Blocks Id System /dev/sdc1 1 243201 1953512001 83 Linux Disk /dev/sdd: 1000.2 GB, 1000204886016 bytes 255 heads, 63 sectors/track, 121601 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x6c6df2ab Device Boot Start End Blocks Id System /dev/sdd1 1 121601 976760001 83 Linux Disk /dev/sde: 1000.2 GB, 1000204886016 bytes 255 heads, 63 sectors/track, 121601 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x6c6df2a1 Device Boot Start End Blocks Id System /dev/sde1 1 121601 976760001 83 Linux Disk /dev/sdf: 1500.3 GB, 1500301910016 bytes 255 heads, 63 sectors/track, 182401 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x2f54e55e Device Boot Start End Blocks Id System /dev/sdf1 1 182401 1465136001 83 Linux Disk /dev/sdg: 2000.4 GB, 2000398934016 bytes 255 heads, 63 sectors/track, 243201 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x00085c3f Device Boot Start End Blocks Id System /dev/sdg1 1 243201 1953512001 83 Linux Disk /dev/dm-0: 1993.9 GB, 1993892429824 bytes 255 heads, 63 sectors/track, 242410 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x00000000 Disk /dev/dm-0 doesn't contain a valid partition table Disk /dev/dm-1: 6291 MB, 6291456000 bytes 255 heads, 63 sectors/track, 764 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x00000000 Disk /dev/dm-1 doesn't contain a valid partition table Disk /dev/sdh: 1000.2 GB, 1000204886016 bytes 255 heads, 63 sectors/track, 121601 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x6c6df17d Device Boot Start End Blocks Id System /dev/sdh1 1 121601 976760001 83 Linux Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_linuxserver-lv_root 1.8T 1.5T 302G 83% / tmpfs 2.0G 420K 2.0G 1% /dev/shm /dev/sda1 194M 40M 144M 22% /boot /dev/sdf1 1.4T 1.2T 158G 88% /var/hda/files/drives/drive3 /dev/sdd1 917G 714G 158G 82% /var/hda/files/drives/drive4 /dev/sde1 917G 715G 157G 83% /var/hda/files/drives/drive5 /dev/sdc1 1.8T 1.6T 157G 91% /var/hda/files/drives/drive6 /dev/sdb1 917G 714G 158G 82% /var/hda/files/drives/drive7 /dev/sdg1 1.8T 1.6T 158G 91% /var/hda/files/drives/drive9 /dev/sdh1 917G 714G 158G 82% /var/hda/files/drives/drive11 funionfs 1.8T 1.5T 302G 83% /var/hda/apps/ushare/files Greyhole Statistics =================== Storage Pool Total - Used = Free + Attic = Possible /var/hda/files/drives/drive3/gh: 1375G - 1148G = 157G + 0G = 157G /var/hda/files/drives/drive5/gh: 917G - 714G = 156G + 0G = 156G /var/hda/files/drives/drive6/gh: 1834G - 1584G = 157G + 0G = 157G /var/hda/files/drives/drive7/gh: 917G - 713G = 157G + 0G = 157G /var/hda/files/drives/drive9/gh: 1834G - 1583G = 157G + 0G = 157G /var/hda/files/drives/drive11/gh: 917G - 713G = 157G + 0G = 157G /var/hda/files/drives/drive4/gh: 917G - 713G = 157G + 0G = 157G

Code: Select all

mysql -u root -phda -e "select * from disk_pool_partitions" hda_production +----+-------------------------------+--------------+---------------------+---------------------+ | id | path | minimum_free | created_at | updated_at | +----+-------------------------------+--------------+---------------------+---------------------+ | 3 | /var/hda/files/drives/drive3 | 10 | 2010-10-24 23:47:44 | 2010-10-24 23:47:44 | | 5 | /var/hda/files/drives/drive5 | 10 | 2010-11-05 04:20:13 | 2010-11-05 04:20:13 | | 6 | /var/hda/files/drives/drive6 | 10 | 2010-11-11 06:04:37 | 2010-11-11 06:04:37 | | 7 | /var/hda/files/drives/drive7 | 10 | 2010-11-14 23:47:31 | 2010-11-14 23:47:31 | | 8 | /var/hda/files/drives/drive9 | 10 | 2011-01-12 03:59:29 | 2011-01-12 03:59:29 | | 12 | /var/hda/files/drives/drive11 | 10 | 2011-01-20 06:40:32 | 2011-01-20 06:40:32 | | 14 | /var/hda/files/drives/drive4 | 10 | 2011-03-10 05:57:49 | 2011-03-10 05:57:49 | +----+-------------------------------+--------------+---------------------+---------------------+

Code: Select all

mysql -u root -phda -e "select concat(path, '/gh') from disk_pool_partitions" hda_production | grep -v 'concat(' | xargs ls -la | fpaste http://fpaste.org/0Mus/

Code: Select all

greyhole --view-queue Greyhole Work Queue Statistics ============================== This table gives you the number of pending operations queued for the Greyhole daemon, per share. Write Delete Rename Books 0 0 0 Completed 0 0 0 Docs 494 0 0 Movies 617 0 0 Music 0 0 0 Pictures 0 0 0 SendToLaptop 0 0 0 SendToPhone 0 0 0 SendToTran 0 0 0 Television 1041 0 0 Torrentflux-b4rt 0 0 0 ================ Total 2152 + 0 + 0 = 2152

Code: Select all

greyhole --debug <one of my missing files> http://fpaste.org/Ype0/

[cpg]

Hmm ..

Code: Select all

Moving this file wouldn't help balancing available space. Skipping.

I have not see this one before ... sorry. we'll have to wait for gboudreau to analyze a bit ...

[gboudreau]

Your Greyhole version is pretty old. You should upgrade.

About your disappearing file:
Nurse Jackie/Season 2/nurse.jackie.s02e02.dvdrip.xvid-reward.sample.avi

That file seems fine. It hasn't been 'seen' by greyhole since Mar. 25, and then, it was when you issued a --balance command.
It's as if the Greyhole daemon is not running. If that file had been scanned by fsck, it would appear in the output of the --debug command, and it's not. What share is that file on ? Is that share using the pool (it should appear in /etc/greyhole.conf) ?

I suggest you upgrade to Amahi 6, since this will also update your Greyhole to the latest version, and should restart the daemon if it wasn't running.

[mattrountree]

I'm hesitant to upgrade since I just took a look at all the missing (beta) apps from Amahi 5, but if that's my best course of action I'll bite the bullet.

I have run fsck manually probably once a day for two weeks. It always says something like scheduling fsck to run once other commands are completed. I wasn't sure if that was related to why greyhole -q only shows a greater and greater number of files in the queue.

[gboudreau]

You can try to fix on Amahi5 if you prefer.

Start by upgrading Greyhole to the latest version (all the following commands should be executed as root):

Code: Select all

rpm -Uvh http://www.greyhole.net/releases/hda-greyhole-0.9.2-1.`uname -m`.rpm

Once that's done, check if the Greyhole service is running:

Code: Select all

service greyhole status

If it isn't for some reason, try to start it:

Code: Select all

service greyhole start

They verify if it's running again.
Look at the greyhole.log for hints, if it says it started, but s subsequent status says it's not running anymore:

Code: Select all

tail -f /var/log/greyhole.log

If Greyhole works as it should, let it run for a while. With all the queued operations (and fsck), it might take a while until it's done.
Monitor greyhole -q or /var/log/greyhole.log to see when it's done.

[Sparticuz]

I'm having the same exact problem, except my queue isn't growning. It's staying at zero. Getting kinda freaked out! Help!

Here is an pastbin of a debugged file that just disappeared.

http://pastebin.com/CewMEFjm

Code: Select all

2.6.35.11-83.fc14.x86_64 samba-3.5.8-74.fc14.x86_64 hda-greyhole-0.9.2-1.x86_64

[gboudreau]

Sparticuz: open a new thread, and make sure to post the complete output of all debugging information that you get when you follow the instructions in the sticky post in this form.

[mattrountree]

Ok, I'm going to take your advice for fixing with Amahi5. I'm having trouble upgrading to Amahi6. Looks like an issue with the Atom 510 chipset on the MSI nettop I'm ussing, but that's an something I'll deal with later and separately...

[mattrountree]

The greyhole update went smoothly. It has been running continuously for 4 days now. The queue was over 150K, but has been steadily dropping. I think at the current rate it may be 20+ days before it is done. I'll update again when it is done.

[mattrountree]

The fsck has finished and the files are still missing. I'm reposting with updated info.

Code: Select all

uname -r; rpm -q samba hda-greyhole 2.6.32.26-175.fc12.x86_64 samba-3.4.9-60.fc12.x86_64 hda-greyhole-0.9.2-1.x86_64

Code: Select all

yum -y install fpaste; fpaste /etc/samba/smb.conf; fpaste /etc/greyhole.conf http://fpaste.org/iZWb/ http://fpaste.org/umyy/

Code: Select all

mount; fdisk -l; df -h; greyhole --stats /dev/mapper/vg_linuxserver-lv_root on / type ext4 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw) /dev/sda1 on /boot type ext4 (rw) /dev/sdf1 on /var/hda/files/drives/drive3 type ext4 (rw) /dev/sdd1 on /var/hda/files/drives/drive4 type ext4 (rw) /dev/sde1 on /var/hda/files/drives/drive5 type ext4 (rw) /dev/sdc1 on /var/hda/files/drives/drive6 type ext4 (rw) /dev/sdb1 on /var/hda/files/drives/drive7 type ext4 (rw) /dev/sdg1 on /var/hda/files/drives/drive9 type ext4 (rw) /dev/sdh1 on /var/hda/files/drives/drive11 type ext4 (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) funionfs on /var/hda/apps/ushare/files type fuse.funionfs (rw,nosuid,nodev,allow_other) gvfs-fuse-daemon on /home/matt/.gvfs type fuse.gvfs-fuse-daemon (rw,nosuid,nodev,user=matt) Disk /dev/sda: 2000.4 GB, 2000398934016 bytes 255 heads, 63 sectors/track, 243201 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x00098578 Device Boot Start End Blocks Id System /dev/sda1 * 1 26 204800 83 Linux Partition 1 does not end on cylinder boundary. /dev/sda2 26 243201 1953307201 8e Linux LVM Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes 255 heads, 63 sectors/track, 121601 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x6c6df128 Device Boot Start End Blocks Id System /dev/sdb1 1 121601 976760001 83 Linux Disk /dev/sdc: 2000.4 GB, 2000398934016 bytes 255 heads, 63 sectors/track, 243201 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x8f41220a Device Boot Start End Blocks Id System /dev/sdc1 1 243201 1953512001 83 Linux Disk /dev/sdd: 1000.2 GB, 1000204886016 bytes 255 heads, 63 sectors/track, 121601 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x6c6df2ab Device Boot Start End Blocks Id System /dev/sdd1 1 121601 976760001 83 Linux Disk /dev/sde: 1000.2 GB, 1000204886016 bytes 255 heads, 63 sectors/track, 121601 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x6c6df2a1 Device Boot Start End Blocks Id System /dev/sde1 1 121601 976760001 83 Linux Disk /dev/sdf: 1500.3 GB, 1500301910016 bytes 255 heads, 63 sectors/track, 182401 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x2f54e55e Device Boot Start End Blocks Id System /dev/sdf1 1 182401 1465136001 83 Linux Disk /dev/sdg: 2000.4 GB, 2000398934016 bytes 255 heads, 63 sectors/track, 243201 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x00085c3f Device Boot Start End Blocks Id System /dev/sdg1 1 243201 1953512001 83 Linux Disk /dev/dm-0: 1993.9 GB, 1993892429824 bytes 255 heads, 63 sectors/track, 242410 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x00000000 Disk /dev/dm-0 doesn't contain a valid partition table Disk /dev/dm-1: 6291 MB, 6291456000 bytes 255 heads, 63 sectors/track, 764 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x00000000 Disk /dev/dm-1 doesn't contain a valid partition table Disk /dev/sdh: 1000.2 GB, 1000204886016 bytes 255 heads, 63 sectors/track, 121601 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x6c6df17d Device Boot Start End Blocks Id System /dev/sdh1 1 121601 976760001 83 Linux Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_linuxserver-lv_root 1.8T 989G 747G 57% / tmpfs 2.0G 420K 2.0G 1% /dev/shm /dev/sda1 194M 74M 110M 41% /boot /dev/sdf1 1.4T 1.3T 75G 95% /var/hda/files/drives/drive3 /dev/sdd1 917G 797G 75G 92% /var/hda/files/drives/drive4 /dev/sde1 917G 797G 75G 92% /var/hda/files/drives/drive5 /dev/sdc1 1.8T 1.7T 72G 96% /var/hda/files/drives/drive6 /dev/sdb1 917G 797G 74G 92% /var/hda/files/drives/drive7 /dev/sdg1 1.8T 1.7T 75G 96% /var/hda/files/drives/drive9 /dev/sdh1 917G 797G 75G 92% /var/hda/files/drives/drive11 funionfs 1.8T 989G 747G 57% /var/hda/apps/ushare/files Greyhole Statistics =================== Storage Pool Total - Used = Free + Attic = Possible /var/hda/files/drives/drive3/gh: 1375G - 1231G = 74G + 0G = 74G /var/hda/files/drives/drive5/gh: 917G - 796G = 74G + 0G = 74G /var/hda/files/drives/drive6/gh: 1834G - 1669G = 72G + 0G = 72G /var/hda/files/drives/drive7/gh: 917G - 796G = 74G + 0G = 74G /var/hda/files/drives/drive9/gh: 1834G - 1666G = 74G + 1G = 75G /var/hda/files/drives/drive11/gh: 917G - 796G = 74G + 0G = 74G /var/hda/files/drives/drive4/gh: 917G - 796G = 74G + 0G = 74G ========================================== Total: 8710G - 7751G = 517G + 1G = 518G

Code: Select all

mysql -u root -phda -e "select * from disk_pool_partitions" hda_production +----+-------------------------------+--------------+---------------------+---------------------+ | id | path | minimum_free | created_at | updated_at | +----+-------------------------------+--------------+---------------------+---------------------+ | 3 | /var/hda/files/drives/drive3 | 10 | 2010-10-24 23:47:44 | 2010-10-24 23:47:44 | | 5 | /var/hda/files/drives/drive5 | 10 | 2010-11-05 04:20:13 | 2010-11-05 04:20:13 | | 6 | /var/hda/files/drives/drive6 | 10 | 2010-11-11 06:04:37 | 2010-11-11 06:04:37 | | 7 | /var/hda/files/drives/drive7 | 10 | 2010-11-14 23:47:31 | 2010-11-14 23:47:31 | | 8 | /var/hda/files/drives/drive9 | 10 | 2011-01-12 03:59:29 | 2011-01-12 03:59:29 | | 12 | /var/hda/files/drives/drive11 | 10 | 2011-01-20 06:40:32 | 2011-01-20 06:40:32 | | 14 | /var/hda/files/drives/drive4 | 10 | 2011-03-10 05:57:49 | 2011-03-10 05:57:49 | +----+-------------------------------+--------------+---------------------+---------------------+

Code: Select all

mysql -u root -phda -e "select concat(path, '/gh') from disk_pool_partitions" hda_production | grep -v 'concat(' | xargs ls -la | fpaste http://fpaste.org/ABbk/

Code: Select all

greyhole --view-queue Greyhole Work Queue Statistics ============================== This table gives you the number of pending operations queued for the Greyhole daemon, per share. Write Delete Rename Books 0 0 0 Completed 0 0 0 Docs 32 0 0 Movies 0 0 0 Music 0 0 0 Pictures 0 0 0 SendToLaptop 0 0 0 SendToPhone 0 0 0 SendToTran 0 0 0 Television 0 0 0 Torrentflux-b4rt 0 0 0 ================ Total 32 + 0 + 0 = 32 The following is the number of pending operations that the Greyhole daemon still needs to parse. Until it does, the nature of those operations is unknown. Spooled operations that have been parsed will be listed above and disappear from the count below. Spooled 421

Code: Select all

greyhole --debug /var/hda/files/Television/Nurse\ Jackie/Season\ 2/nurse.jackie.s02e02.dvdrip.xvid-reward.sample.avi Debugging file operations for file named "/var/hda/files/Television/Nurse Jackie/Season 2/nurse.jackie.s02e02.dvdrip.xvid-reward.sample.avi" From DB ======= From logs ========= From filesystem =============== Landing Zone: ls: cannot access /hda/files/Television/Nurse Jackie/Season 2/nurse.jackie.s02e02.dvdrip.xvid-reward.sample.avi: No such file or directory Graveyard: File copies:

Code: Select all

find -name nurse.jackie.s02e02.dvdrip.xvid-reward.sample.avi ./drive9/gh/Television/Nurse Jackie/Season 2/nurse.jackie.s02e02.dvdrip.xvid-reward.sample.avi ./drive9/gh/.gh_graveyard/Television/Nurse Jackie/Season 2/nurse.jackie.s02e02.dvdrip.xvid-reward.sample.avi