(some) Deleted files are restored after the weekly fsck

[kikkegek]

A while ago I added a terabyte drive to my storage pool to increase storage. I also moved the landing zone to a different location. Ofcourse I followed the detailed Wiki instructions for doing so. Everything seemed fine for a little while. but recently I noticed that files I had deleted seem to return after the weekly FSCK job.

here is a partial report of that weekly job:

fsck_files of Greyhole shares on localhost.localdomain
AMAHI HDA
x

root <jochenz@gmail.com>
29 jul. (1 dag geleden)

aan root
fsck report
-----------
Scanned directory: Missing files
Started: 2014-07-06 03:34:09
Ended: 2014-07-29 10:58:05
Duration: 559h 23m 56s
Metadata Store:
Found 0 directories
Found 3,026 files
Landing Zone (shares):
Found 0 directories
Found 1,665 files
Found 0 orphans
Trash size:
/var/hda/files/drives/drive1/gh/.gh_trash = 13M
/var/hda/files/drives/drive2/gh/.gh_trash = 20M
/var/hda/files/drives/drive3/gh/.gh_trash = 101G

Problems:
Found 114 files in the metadata store for which no file copies were found.
Those files were removed from the Landing Zone. (i.e. those files are now gone!) They will re-appear in your shares if a copy re-appear and fsck is run.
If you don't want to see those files listed here each time fsck runs, delete the corresponding files from the metadata store using "greyhole --delete-metadata='<path>'", where <path> is one of the value listed below.
Files with no copies:
Downloads/THE.RUM.DIARY.2011.1080P.CUSTOM.NL.SRT/The Rum Diary (2011) X264 1080P SimplyReleases Toppers/The Rum Diary (2011) X264 1080P SimplyReleases Toppers.mkv
Movies/.Trash-1000/files/88 Minutes (2007)/88.minutes.2007.dvdrip.xvid.cd1-tdl.avi
Movies/.Trash-1000/files/88 Minutes (2007)/88.minutes.2007.dvdrip.xvid.cd2-tdl.avi
Movies/DOCUMENTARIES/xstreem-afterlame/xstreem-afterlame-d.nzb
Movies/DOCUMENTARIES/xstreem-afterlame/xstreem-afterlame-d.sfv

[/quote]

here is the required info from the troubleshooting guide:

[jochen@localhost ~]$ uname -r; rpm -q samba amahi-greyhole
3.10.3-300.fc19.x86_64
samba-4.0.7-2.fc19.x86_64
amahi-greyhole-0.9.60-1.x86_64

smb.conf
http://ur1.ca/hvhd6

greyhole.conf
http://ur1.ca/hvhd9

mount:

[root@localhost jochen]# mount
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,size=1014764k,nr_inodes=253691,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
/dev/sdc5 on / type ext4 (rw,relatime,data=ordered)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=34,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
tmpfs on /tmp type tmpfs (rw)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/sdc1 on /boot type ext4 (rw,relatime,data=ordered)
/dev/sda1 on /var/hda/files/drives/drive2 type ext4 (rw,relatime,data=ordered)
/dev/sdd1 on /var/hda/files/drives/drive3 type ext4 (rw,relatime,data=ordered)
/dev/sdb1 on /var/hda/files/drives/drive1 type ext4 (rw,relatime,data=ordered)
none on /var/spool/greyhole/mem type tmpfs (rw,relatime,size=4096k)
//127.0.0.1/Docs on /mnt/samba/Docs type cifs (rw,relatime,vers=1.0,sec=ntlmssp,cache=strict,username=jochen,domain=HOME,uid=1000,forceuid,gid=100,for cegid,addr=127.0.0.1,file_mode=0660,dir_mode=0770,hard,nounix,nobrl,mfsymlinks,rsize=61440,wsize=65536,actimeo=1)
//127.0.0.1/Downloads on /mnt/samba/Downloads type cifs (rw,relatime,vers=1.0,sec=ntlmssp,cache=strict,username=jochen,domain=HOME,uid=1000,forceuid,g id=100,forcegid,addr=127.0.0.1,file_mode=0660,dir_mode=0770,hard,nounix,nobrl,mfsymlinks,rsize=61440,wsize=65536,actimeo=1)
//127.0.0.1/Greyhole Trash on /mnt/samba/Greyhole Trash type cifs (rw,relatime,vers=1.0,sec=ntlmssp,cache=strict,username=jochen,domain=HOME,uid=1000, forceuid,gid=100,forcegid,addr=127.0.0.1,file_mode=0660,dir_mode=0770,hard,nounix,nobrl,mfsymlinks,rsize=61440,wsize=65536,actimeo=1)
//127.0.0.1/Movies on /mnt/samba/Movies type cifs (rw,relatime,vers=1.0,sec=ntlmssp,cache=strict,username=jochen,domain=HOME,uid=1000,forceuid,gid=100 ,forcegid,addr=127.0.0.1,file_mode=0660,dir_mode=0770,hard,nounix,nobrl,mfsymlinks,rsize=61440,wsize=65536,actimeo=1)
//127.0.0.1/Music on /mnt/samba/Music type cifs (rw,relatime,vers=1.0,sec=ntlmssp,cache=strict,username=jochen,domain=HOME,uid=1000,forceuid,gid=100,f orcegid,addr=127.0.0.1,file_mode=0660,dir_mode=0770,hard,nounix,nobrl,mfsymlinks,rsize=61440,wsize=65536,actimeo=1)
//127.0.0.1/Newsgroups on /mnt/samba/Newsgroups type cifs (rw,relatime,vers=1.0,sec=ntlmssp,cache=strict,username=jochen,domain=HOME,uid=1000,forceuid ,gid=100,forcegid,addr=127.0.0.1,file_mode=0660,dir_mode=0770,hard,nounix,nobrl,mfsymlinks,rsize=61440,wsize=65536,actimeo=1)
//127.0.0.1/Pictures on /mnt/samba/Pictures type cifs (rw,relatime,vers=1.0,sec=ntlmssp,cache=strict,username=jochen,domain=HOME,uid=1000,forceuid,gid =100,forcegid,addr=127.0.0.1,file_mode=0660,dir_mode=0770,hard,nounix,nobrl,mfsymlinks,rsize=61440,wsize=65536,actimeo=1)
//127.0.0.1/Software on /mnt/samba/Software type cifs (rw,relatime,vers=1.0,sec=ntlmssp,cache=strict,username=jochen,domain=HOME,uid=1000,forceuid,gid =100,forcegid,addr=127.0.0.1,file_mode=0660,dir_mode=0770,hard,nounix,nobrl,mfsymlinks,rsize=61440,wsize=65536,actimeo=1)
//127.0.0.1/Torrents on /mnt/samba/Torrents type cifs (rw,relatime,vers=1.0,sec=ntlmssp,cache=strict,username=jochen,domain=HOME,uid=1000,forceuid,gid =100,forcegid,addr=127.0.0.1,file_mode=0660,dir_mode=0770,hard,nounix,nobrl,mfsymlinks,rsize=61440,wsize=65536,actimeo=1)

fdisk -l

[root@localhost jochen]# fdisk -l

Disk /dev/sda: 1000.2 GB, 1000204886016 bytes, 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk label type: dos
Disk identifier: 0x7f04484c

Device Boot Start End Blocks Id System
/dev/sda1 63 1953525167 976762552+ 83 Linux
Partition 1 does not start on physical sector boundary.

Disk /dev/sdb: 750.2 GB, 750156374016 bytes, 1465149168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x9310de50

Device Boot Start End Blocks Id System
/dev/sdb1 2048 1465147391 732572672 83 Linux

Disk /dev/sdc: 40.0 GB, 40018599936 bytes, 78161328 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00069118

Device Boot Start End Blocks Id System
/dev/sdc1 * 2048 411647 204800 83 Linux
/dev/sdc2 411648 4638719 2113536 82 Linux swap / Solaris
/dev/sdc3 4638720 8865791 2113536 82 Linux swap / Solaris
/dev/sdc4 8865792 78161327 34647768 5 Extended
/dev/sdc5 8867840 78159871 34646016 83 Linux

Disk /dev/sdd: 2000.4 GB, 2000398934016 bytes, 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk label type: dos
Disk identifier: 0x00000000

Device Boot Start End Blocks Id System
/dev/sdd1 63 3907029167 1953514552+ 83 Linux
Partition 1 does not start on physical sector boundary.

df -h

[root@localhost jochen]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sdc5 33G 11G 21G 35% /
devtmpfs 991M 0 991M 0% /dev
tmpfs 997M 0 997M 0% /dev/shm
tmpfs 997M 548K 997M 1% /run
tmpfs 997M 0 997M 0% /sys/fs/cgroup
tmpfs 997M 24K 997M 1% /tmp
/dev/sdc1 190M 58M 118M 33% /boot
/dev/sda1 917G 351G 520G 41% /var/hda/files/drives/drive2
/dev/sdd1 1.8T 1.1T 710G 60% /var/hda/files/drives/drive3
/dev/sdb1 688G 190G 463G 30% /var/hda/files/drives/drive1
none 4.0M 0 4.0M 0% /var/spool/greyhole/mem
//127.0.0.1/Docs 3.4T 1.8T 1.7T 51% /mnt/samba/Docs
//127.0.0.1/Downloads 3.4T 1.8T 1.7T 51% /mnt/samba/Downloads
//127.0.0.1/Greyhole Trash 3.4T 1.8T 1.7T 51% /mnt/samba/Greyhole Trash
//127.0.0.1/Movies 3.4T 1.8T 1.7T 51% /mnt/samba/Movies
//127.0.0.1/Music 3.4T 1.8T 1.7T 51% /mnt/samba/Music
//127.0.0.1/Pictures 3.4T 1.8T 1.7T 51% /mnt/samba/Pictures
//127.0.0.1/Software 3.4T 1.8T 1.7T 51% /mnt/samba/Software
//127.0.0.1/Torrents 917G 398G 520G 44% /mnt/samba/Torrents

greyhole --stats

[root@localhost jochen]# greyhole --stats

Greyhole Statistics
===================

Storage Pool
Total - Used = Free + Trash = Possible
/var/hda/files/drives/drive1/gh: 688G - 190G = 463G + 0G = 463G
/var/hda/files/drives/drive2/gh: 917G - 351G = 520G + 0G = 520G
/var/hda/files/drives/drive3/gh: 1834G - 1031G = 710G + 0G = 710G
==========================================
Total: 3438G - 1571G = 1692G + 0G = 1692G

listing drives in storage pool gives an error

[root@localhost jochen]# mysql -u root -phda -e "select * from disk_pool_partitions" hda_production
ERROR 1146 (42S02) at line 1: Table 'hda_production.disk_pool_partitions' doesn't exist

again an error at listing the root directories of each drive

[root@localhost jochen]# mysql -u root -phda -e "select concat(path, '/gh') from disk_pool_partitions" hda_production | grep -v 'concat(' | xargs ls -la | fpaste
ERROR 1146 (42S02) at line 1: Table 'hda_production.disk_pool_partitions' doesn't exist
Uploading (5.0KiB)...
http://ur1.ca/hvhj2 -> http://paste.fedoraproject.org/122032/36238140

greyhole work queue:

[root@localhost jochen]# greyhole --view-queue

Greyhole Work Queue Statistics
==============================

This table gives you the number of pending operations queued for the Greyhole daemon, per share.

Write Delete Rename Check
Docs 0 0 0 0
Downloads 0 0 0 0
Movies 0 0 0 0
Music 0 0 0 0
Newsgroups 0 0 0 0
Pictures 0 0 0 0
Software 0 0 0 0
Torrents 218 0 0 114
==============================================
Total 218 0 0 114
Write Delete Rename Check

The following is the number of pending operations that the Greyhole daemon still needs to parse.
Until it does, the nature of those operations is unknown.
Spooled operations that have been parsed will be listed above and disappear from the count below.

Spooled 0

well, hope this is enough for now. hope you guys can help me out here.

[bigfoot65]

Why is Greyhole Trash listed in your greyhole.conf file? Although Greyhole should ignore it, I would remove from the file. It's not needed.

The portion about listing drives in storage pool and he root directories of each drive gives an error is a warning sign something may not be right. Recommend you use something like Adminer or phpMyAdmin from the Amahi app store to verify if the table exists and is populated.

This might have something to do with your problem.

[kikkegek]

ok, thanks.

I installed PHPMyAdmin but I have no idea where to start looking.

[bigfoot65]

Look for disk_pool_partitions table in hda_production to start with. That is one of the errors you reflected in your initial post.