Data in shares missing (or not displaying)

[jbmia]

So, I've migrated a bunch of data using rsync ( rsync -avzru oldwhsshare newamahishare) into a 7 drive pool and all seemed to be fine.. I just finalized the last drive and now I notice that my shared folders (movies, docs, books, etc) don't show any files. Directories are displayed, but no actual data. When I browse to some of the drivex/gh folders I can see actual data, but it's not displayed in my shares (either mounted locally in /mnt/samba/sharenamex or remotely via \\HDA\sharenamex. Any ideas? Here's my details:



1. What version of Fedora, Samba & Greyhole are you running?
Code:
uname -r; rpm -q samba hda-greyhole

Code: Select all

6.31.5-127.fc12.i686.PAE samba-3.4.2-47.fc12.i686 hda-greyhole-0.7.5-1.i386

2. The content of the /etc/samba/smb.conf & /etc/greyhole.conf files (provide paste URLs):
Code:
yum -y install fpaste; fpaste /etc/samba/smb.conf; fpaste /etc/greyhole.conf

Code: Select all

http://fpaste.org/7cnm/ http://fpaste.org/n2vq/

3. The result of the following commands:
Code:
mount; fdisk -l; df -h; greyhole --stats

Code: Select all

/dev/mapper/VolGroup-lv_root on / type ext4 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw) /dev/sdb1 on /boot type ext4 (rw) /dev/sdd1 on /var/hda/files/drives/drive1 type ext4 (rw) /dev/sda1 on /var/hda/files/drives/drive2 type ext4 (rw) /dev/sdc1 on /var/hda/files/drives/drive3 type ext4 (rw) /dev/sdf1 on /var/hda/files/drives/drive4 type ext4 (rw) /dev/sde1 on /var/hda/files/drives/drive5 type ext4 (rw) /dev/sdh1 on /var/hda/files/drives/drive6 type ext4 (rw) /dev/sdg1 on /var/hda/files/drives/drive7 type ext4 (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) //127.0.0.1/movies on /mnt/samba/movies type cifs (rw,mand) //127.0.0.1/movies_home on /mnt/samba/movies_home type cifs (rw,mand) //127.0.0.1/music on /mnt/samba/music type cifs (rw,mand) //127.0.0.1/pictures on /mnt/samba/pictures type cifs (rw,mand) //127.0.0.1/public on /mnt/samba/public type cifs (rw,mand) //127.0.0.1/software on /mnt/samba/software type cifs (rw,mand) //127.0.0.1/users on /mnt/samba/users type cifs (rw,mand) //127.0.0.1/videos on /mnt/samba/videos type cifs (rw,mand) Disk /dev/sda: 1000.2 GB, 1000204886016 bytes 255 heads, 63 sectors/track, 121601 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x4bafc7b3 Device Boot Start End Blocks Id System /dev/sda1 1 121602 976760832 83 Linux Disk /dev/sdb: 250.1 GB, 250059350016 bytes 255 heads, 63 sectors/track, 30401 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0xd16ce3ee Device Boot Start End Blocks Id System /dev/sdb1 * 1 26 204800 83 Linux Partition 1 does not end on cylinder boundary. /dev/sdb2 26 30401 243991201 8e Linux LVM Disk /dev/sdc: 1000.2 GB, 1000204886016 bytes 255 heads, 63 sectors/track, 121601 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0xc87ee887 Device Boot Start End Blocks Id System /dev/sdc1 1 121602 976760832 83 Linux Disk /dev/sdd: 300.1 GB, 300090728448 bytes 255 heads, 63 sectors/track, 36483 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0xd16ce393 Device Boot Start End Blocks Id System /dev/sdd1 1 36484 293055488 83 Linux Disk /dev/sde: 1500.3 GB, 1500301910016 bytes 255 heads, 63 sectors/track, 182401 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x18249b79 Device Boot Start End Blocks Id System /dev/sde1 1 182402 1465137152 83 Linux Disk /dev/sdf: 1000.2 GB, 1000204886016 bytes 255 heads, 63 sectors/track, 121601 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x5c2ff997 Device Boot Start End Blocks Id System /dev/sdf1 1 121602 976760832 83 Linux Disk /dev/sdg: 2000.4 GB, 2000398934016 bytes 255 heads, 63 sectors/track, 243201 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x29cd4bd0 Device Boot Start End Blocks Id System /dev/sdg1 1 243202 1953513472 83 Linux Disk /dev/sdh: 250.1 GB, 250059350016 bytes 255 heads, 63 sectors/track, 30401 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0xd16ce398 Device Boot Start End Blocks Id System /dev/sdh1 1 30402 244197376 83 Linux Disk /dev/dm-0: 244.3 GB, 244259487744 bytes 255 heads, 63 sectors/track, 29696 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x00000000 Disk /dev/dm-0 doesn't contain a valid partition table Disk /dev/dm-1: 5586 MB, 5586812928 bytes 255 heads, 63 sectors/track, 679 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x00000000 Disk /dev/dm-1 doesn't contain a valid partition table Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup-lv_root 224G 52G 162G 25% / tmpfs 1.9G 88K 1.9G 1% /dev/shm /dev/sdb1 194M 22M 163M 12% /boot /dev/sdd1 276G 262G 0 100% /var/hda/files/drives/drive1 /dev/sda1 917G 871G 0 100% /var/hda/files/drives/drive2 /dev/sdc1 917G 441G 431G 51% /var/hda/files/drives/drive3 /dev/sdf1 917G 366G 506G 42% /var/hda/files/drives/drive4 /dev/sde1 1.4T 265G 1.1T 21% /var/hda/files/drives/drive5 /dev/sdh1 230G 188M 218G 1% /var/hda/files/drives/drive6 /dev/sdg1 1.8T 24G 1.7T 2% /var/hda/files/drives/drive7 //127.0.0.1/movies 6.4T 2.5T 3.9T 40% /mnt/samba/movies //127.0.0.1/movies_home 6.4T 2.5T 3.9T 40% /mnt/samba/movies_home //127.0.0.1/music 6.4T 2.5T 3.9T 40% /mnt/samba/music //127.0.0.1/pictures 6.4T 2.5T 3.9T 40% /mnt/samba/pictures //127.0.0.1/public 6.4T 2.5T 3.9T 40% /mnt/samba/public //127.0.0.1/software 6.4T 2.5T 3.9T 40% /mnt/samba/software //127.0.0.1/users 6.4T 2.5T 3.9T 40% /mnt/samba/users //127.0.0.1/videos 6.4T 2.5T 3.9T 40% /mnt/samba/videos Greyhole Statistics =================== Storage Pool Total - Used = Free + Attic = Possible /var/hda/files/drives/drive1/gh: 275G - 261G = 0G + 0G = 0G /var/hda/files/drives/drive2/gh: 917G - 871G = 0G + 22G = 22G /var/hda/files/drives/drive3/gh: 917G - 440G = 430G + 0G = 430G /var/hda/files/drives/drive4/gh: 917G - 365G = 505G + 33G = 538G /var/hda/files/drives/drive5/gh: 1375G - 265G = 1041G + 0G = 1041G /var/hda/files/drives/drive6/gh: 229G - 0G = 217G + 0G = 217G /var/hda/files/drives/drive7/gh: 1834G - 24G = 1717G + 0G = 1717G

4. The list drives in your storage pool (per Amahi platform):
Code:
mysql -u root -phda -e "select * from disk_pool_partitions" hda_production

Code: Select all

+----+------------------------------+--------------+---------------------+---------------------+ | id | path | minimum_free | created_at | updated_at | +----+------------------------------+--------------+---------------------+---------------------+ | 1 | /var/hda/files/drives/drive1 | 10 | 2010-12-08 04:50:14 | 2010-12-08 04:50:14 | | 2 | /var/hda/files/drives/drive2 | 10 | 2010-12-12 15:59:29 | 2010-12-12 15:59:29 | | 3 | /var/hda/files/drives/drive3 | 10 | 2010-12-15 04:28:31 | 2010-12-15 04:28:31 | | 4 | /var/hda/files/drives/drive4 | 10 | 2010-12-16 04:13:43 | 2010-12-16 04:13:43 | | 5 | /var/hda/files/drives/drive5 | 10 | 2010-12-16 20:29:36 | 2010-12-16 20:29:36 | | 6 | /var/hda/files/drives/drive6 | 10 | 2010-12-16 22:55:38 | 2010-12-16 22:55:38 | | 7 | /var/hda/files/drives/drive7 | 10 | 2010-12-17 16:10:30 | 2010-12-17 16:10:30 | +----+------------------------------+--------------+---------------------+---------------------+

5. A list of the directories on the root of the drives included in your storage pool, obtained with the following command (provide a paste URL):
Code:
mysql -u root -phda -e "select concat(path, '/gh') from disk_pool_partitions" hda_production | grep -v 'concat(' | xargs ls -la | fpaste

Code: Select all

http://fpaste.org/S56c/

6. The Greyhole work queue:
Code:
greyhole --view-queue

Code: Select all

Greyhole Work Queue Statistics ============================== This table gives you the number of pending operations queued for the Greyhole daemon, per share. Write Delete Rename Books 0 0 0 Docs 0 0 0 Movies 0 0 0 Movies_Home 0 0 0 Music 4954 0 0 Pictures 10376 0 4594 Public 130 0 65 Software 522 0 261 Users 9744 0 4873 Videos 6 0 4 =========== Total 25732 + 0 + 9797 = 35529

Regards,

John

[gboudreau]

Choose a file that is missing from the share, and do:

Code: Select all

greyhole --debug filename

Post the result in a new message.

[jbmia]

Code: Select all

[john@localhost /]$ greyhole --debug 20001216_029.jpg Debugging file operations for file named "/20001216_029.jpg" From DB ======= [2010-12-15 08:28:44] Task ID 303602: rename Pictures/2000 - 12/.20001216_029.jpg.102rwJ -> Pictures/2000 - 12/20001216_029.jpg [2010-12-16 15:46:28] Task ID 389455: fsck_file Pictures/2000 - 12/20001216_029.jpg From logs ========= Dec 15 14:55:52 7 parse_logs: Done parsing. Dec 15 14:55:52 7 write: Now working on task ID 303601: write Pictures/2000 - 12/.20001216_029.jpg.102rwJ Dec 15 14:55:52 7 write: File write: /var/hda/files/pictures/2000 - 12/.20001216_029.jpg.102rwJ doesn't exist anymore. Skipping. -- Dec 15 14:55:52 7 parse_logs: Done parsing. Dec 15 14:55:52 7 rename: Now working on task ID 303602: rename Pictures/2000 - 12/.20001216_029.jpg.102rwJ -> Pictures/2000 - 12/20001216_029.jpg Dec 15 14:55:52 6 rename: File renamed: /var/hda/files/pictures/2000 - 12/.20001216_029.jpg.102rwJ -> /var/hda/files/pictures/2000 - 12/20001216_029.jpg Dec 15 14:55:53 7 rename: Loading tombstones for Pictures/2000 - 12/.20001216_029.jpg.102rwJ... Got 0 tombstones. Dec 15 14:55:53 7 rename: Loading tombstones for Pictures/2000 - 12/20001216_029.jpg... Got 0 tombstones. Dec 15 14:55:53 7 rename: Loading tombstones for Pictures/2000 - 12/.20001216_029.jpg.102rwJ... Got 0 tombstones. Dec 15 14:55:53 6 rename: File created: Pictures/2000 - 12/20001216_029.jpg - 30.7KB Dec 15 14:55:53 7 rename: Loading tombstones for Pictures/2000 - 12/20001216_029.jpg... Got 0 tombstones. Dec 15 14:55:53 7 rename: Drives with available space: /var/hda/files/drives/drive3/gh (602GB avail) - /var/hda/files/drives/drive1/gh (51.8GB avail) Dec 15 14:55:53 7 rename: Drives full: /var/hda/files/drives/drive2/gh (0.00B free) Dec 15 14:55:53 7 rename: Saving 1 tombstones for Pictures/2000 - 12/20001216_029.jpg Dec 15 14:55:53 7 rename: Saving tombstones in /var/hda/files/drives/drive3/gh/.gh_graveyard/Pictures/2000 - 12/20001216_029.jpg Dec 15 14:55:53 7 rename: Saving tombstones (backup) in /var/hda/files/drives/drive1/gh/.gh_graveyard_backup/Pictures/2000 - 12/20001216_029.jpg Dec 15 14:55:53 7 rename: Copying file to /var/hda/files/drives/drive3/gh/Pictures/2000 - 12/20001216_029.jpg Dec 15 14:55:53 7 rename: Creating symlink in share pointing to the above file copy. Dec 15 14:55:53 7 rename: Saving 1 tombstones for Pictures/2000 - 12/20001216_029.jpg Dec 15 14:55:53 7 rename: Saving tombstones in /var/hda/files/drives/drive3/gh/.gh_graveyard/Pictures/2000 - 12/20001216_029.jpg Dec 15 14:55:53 7 rename: Saving tombstones (backup) in /var/hda/files/drives/drive1/gh/.gh_graveyard_backup/Pictures/2000 - 12/20001216_029.jpg Dec 15 14:55:53 7 parse_logs: Parsing Samba logs... Dec 16 20:54:51 7 parse_logs: Done parsing. Dec 16 20:54:51 7 fsck_file: Now working on task ID 389455: fsck_file Pictures/2000 - 12/20001216_029.jpg Dec 16 20:54:51 7 fsck_file: Found /var/hda/files/drives/drive3/gh/Pictures/2000 - 12/20001216_029.jpg Dec 16 20:54:51 7 fsck_file: Loading tombstones for Pictures/2000 - 12/20001216_029.jpg... Got 1 tombstones. Dec 16 20:54:51 6 fsck_file: Symlink was missing. Creating new symlink. Dec 16 20:54:51 7 fsck_file: Updating symlink at /var/hda/files/pictures/2000 - 12/20001216_029.jpg to point to /var/hda/files/drives/drive3/gh/Pictures/2000 - 12/20001216_029.jpg Dec 16 20:54:51 7 fsck_file: Saving 1 tombstones for Pictures/2000 - 12/20001216_029.jpg Dec 16 20:54:51 7 fsck_file: Saving tombstones in /var/hda/files/drives/drive3/gh/.gh_graveyard/Pictures/2000 - 12/20001216_029.jpg Dec 16 20:54:51 7 fsck_file: Saving tombstones (backup) in /var/hda/files/drives/drive1/gh/.gh_graveyard_backup/Pictures/2000 - 12/20001216_029.jpg Dec 16 20:54:51 7 parse_logs: Parsing Samba logs... From filesystem =============== Landing Zone: ls: cannot access /var/hda/files/pictures/2000 - 12/20001216_029.jpg: No such file or directory Graveyard: -rw-rw-rw- 1 root root 156 2010-12-16 20:54 /var/hda/files/drives/drive3/gh/.gh_graveyard/Pictures/2000 - 12/20001216_029.jpg array ( 0 => stdClass::__set_state(array( 'path' => '/var/hda/files/drives/drive3/gh/Pictures/2000 - 12/20001216_029.jpg', 'is_linked' => true, 'state' => 'OK', )), ) File copies: -rwxrw-r-- 1 john users 31447 2010-12-15 14:55 /var/hda/files/drives/drive3/gh/Pictures/2000 - 12/20001216_029.jpg

[gboudreau]

Something (not Greyhole) seems to have eaten your symlinks.

It was there on Dec 16:

Dec 16 20:54:51 7 fsck_file: Updating symlink at /var/hda/files/pictures/2000 - 12/20001216_029.jpg to point to /var/hda/files/drives/drive3/gh/Pictures/2000 - 12/20001216_029.jpg

But isn't there anymore:

ls: cannot access /var/hda/files/pictures/2000 - 12/20001216_029.jpg: No such file or directory

You can get them back by running:

Code: Select all

greyhole --fsck

Look in /var/log/greyhole.log to see when it's done with the fsck, and all your symlinks should be back.
I don't think Greyhole is the problem here; if it couldn't create the symlink on Dec 16, the log would say so.
And if it had played with it in any way after that, it would also be logged.
Something else deleted those symlinks. Something that deleted the symlinks directly on the HDA itself (not through the shares).
Any idea what it could be?

[jbmia]

Sorry for not getting back to your earlier.. had to step out..

I had already scheduled a greyhole --fsck and now on return, I can already see files being repopulated.

I'm not sure if it's completed, how would I determine that in the log? Is there a phrase I could search for? Here's a snapshot... I see a "PHP Error".. is that of any significance?? :

Code: Select all

Dec 17 17:01:18 7 write: Now working on task ID 704754: write Pictures/2004 - 06/Vegas Trip - Dinner with the Sardinas family 061204/Vegas Trip - Dinner with the Sardinas family 061204 129.jpg Dec 17 17:01:18 6 write: File changed: Pictures/2004 - 06/Vegas Trip - Dinner with the Sardinas family 061204/Vegas Trip - Dinner with the Sardinas family 061204 129.jpg - 1.61MB Dec 17 17:01:18 7 write: Will use source file: /var/hda/files/drives/drive7/gh/Pictures/2004 - 06/Vegas Trip - Dinner with the Sardinas family 061204/Vegas Trip - Dinner with the Sardinas family 061204 129.jpg Dec 17 17:01:18 7 write: Loading tombstones for Pictures/2004 - 06/Vegas Trip - Dinner with the Sardinas family 061204/Vegas Trip - Dinner with the Sardinas family 061204 129.jpg... Got 1 tombstones. Dec 17 17:01:18 7 write: Saving 1 tombstones for Pictures/2004 - 06/Vegas Trip - Dinner with the Sardinas family 061204/Vegas Trip - Dinner with the Sardinas family 061204 129.jpg Dec 17 17:01:18 7 write: Saving tombstones in /var/hda/files/drives/drive7/gh/.gh_graveyard/Pictures/2004 - 06/Vegas Trip - Dinner with the Sardinas family 061204/Vegas Trip - Dinner with the Sardinas family 061204 129.jpg Dec 17 17:01:18 7 write: Saving tombstones (backup) in /var/hda/files/drives/drive1/gh/.gh_graveyard_backup/Pictures/2004 - 06/Vegas Trip - Dinner with the Sardinas family 061204/Vegas Trip - Dinner with the Sardinas family 061204 129.jpg Dec 17 17:01:18 7 write: File copy at /var/hda/files/drives/drive7/gh/Pictures/2004 - 06/Vegas Trip - Dinner with the Sardinas family 061204/Vegas Trip - Dinner with the Sardinas family 061204 129.jpg is already up to date. Dec 17 17:01:18 7 write: More than 90000 tasks queued... Won't queue any more at this time. Dec 17 17:01:18 4 check_pool: PHP Warning [8]: unserialize(): Error at offset 222 of 255 bytes in /usr/bin/greyhole on line 3303

I'm not sure what could have deleted all the tombstones.. I'm no expert and learning as a I go... I did run a test box fora few months, but theres always things that come up.. Some random thoughts... I've been using rsync... both from NTFS drives to my shares and from my oriignal WHS box before I brought it down over to my Amahi test box.. I've all the drives from the old WHS box, live on this Amahi box all in the greyhole storage pool.. The only thing I need to do now is rsyn orphan data that's over on my Amaahi test box to the production box. I'm new to Rsync and at the beginning I was copying files directly to the /var/hda/files/sharex folders, as it was deemed okay to make my initial writes there (no mod or deletes) , but when I discovered the ability to process local share mounts using samba, I changed to rsysnc to that samaba mount instead... At least one time, I forgot a trailing "/" when using rsync and I copied a /movies folder into a /movies folder... s, I rsync the /movies/movies to /movies and then delted the /movies//movies folder from within the samba mount. I also inadvertently ran a "umount -a" that disconnected all my storage drives. temporarily. .while I was getting ready to partition my last drive in ext4 format.. in preparation to add to the pool... Thaats about all I can think off...

Does that error in the gryhole.log look like anything bad??

Thanks for al your assistance... Sorry for the typos.. there seems to e some problem typing in the message body when posting.. really odd behaviour that makes it hard to see what your typing..

[gboudreau]

When Greyhole is done, the log file will say something about sleeping every 10 seconds.
The log above tells me it might be a while until you see it, since you have more than 90k tasks still to be processed.
Also, a fsck is considered a normal file operation, so when you do a --fsck, all you do is add a fsck operation into the queue. If your queue always has a lot of unprocessed entries, it could take a while to actually start the fsck.

I'll have a look at the error at the bottom. Never saw it.
Does it happen often ? (on every file write), or just sometimes ?

[jbmia]

It seems to be writing that error to the log very frequently.. Just about a quickly as I can hit the up arrow on my keyboard to recall the last tail command and hit enter is as frequently as it's being written.. So, yeah, it might be for every transaction..

jbmia

[gboudreau]

Did you migrate to MySQL for Greyhole?
If you didn't, please to now: http://wiki.amahi.org/index.php/Greyhol ... e_to_MySQL

Then:

Code: Select all

mysql -ugreyhole -pgreyhole -e "SELECT value FROM settings WHERE name = 'Gone-FSCKed-Drives'" greyhole

That value seems corrupted, so I'd like to see what value it has in your database.

Thanks.

[jbmia]

Yes, I did migrate to MySQL...

Here are the results of the query you requested...

Code: Select all

[john@localhost ~]$ mysql -ugreyhole -pgreyhole -e "SELECT value FROM settings WHERE name = 'Gone-FSCKed-Drives'" greyhole +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | value | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | a:6:{s:31:"/var/hda/files/drives/drive6/gh";b:1;s:31:"/var/hda/files/drives/drive1/gh";b:1;s:31:"/var/hda/files/drives/drive2/gh";b:1;s:31:"/var/hda/files/drives/drive3/gh";b:1;s:31:"/var/hda/files/drives/drive4/gh";b:1;s:31:"/var/hda/files/drives/drive5/ | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Thanks for all of your assistance...

jbmia

[gboudreau]

To fix:

Code: Select all

mysql -ugreyhole -pgreyhole -e "UPDATE settings SET value='a:0:{}' WHERE name = 'Gone-FSCKed-Drives'" greyhole