Files disappearing from shares, but still exist in pool

[Workdawg]

So, here's the background.

I built my amahi server and put a clean 2tb drive into it. I used the instructions on the wiki for "adding a second drive" using cfdisk, gparted, etc. Everything worked great. I had 2 640gb hard drives with many files on them, roughly 900gb total. I put them into my desktop machine one at a time and copied all the files over to the /movies share to properly add them to the storage pool.

At some point during the first drive, the transfer died because the root drive filled up. I moved the landing zone onto the 2tb drive to prevent the transfer from dieing again. After that, the first drive went fine. All the files moved properly, I tested a handful at random, great. I noticed that even though I moved the landing zone to the 2tb drive, it was still slow. I used iotop to check, and the 2tb drive was still maxing out since the landing zone was on it and the files were being moved to it.

I then took that 640gb drive (the one I had just copied files off of) and added it to the storage pool, again according to the wiki instructions. Right after that, I moved the landing zone onto that drive so that I could spread the io across the two drives. Once that was complete, I started copying files from the second drive over to the storage pool.

After copying the rest over, I confirmed everything was there. Then I pulled the 2nd 640gb drive from my desktop and intended to add it to the storage pool as well. Once I got it hooked up to the amahi machine, I did one last check on the /movies share to make sure all my files were still there before wiping the drive. It turns out, half of my files from the /movies share are missing. Looking at the amahi setup dashboard, it's still showing all of the space on the 2tb drive used, so the files are still there, but the links in the share seem to be broken. I tried a greyhole --fsck but that didn't fix it. I've restarted the machine a couple of times, still to no avail.

Someone on IRC at #amahi helped me a little bit and I was able to discover that all the files are still in the pool... stored in /var/hda/files/drives/drive1/movies, but only half of them (the half in the share) are in /var/hda/files/drives/drive1/gh/movies.

If you can help me recreate the links to the files that are broken, that would be awesome.

Thanks in advance!

---------------
Here is the required info:

1.
Fedora: 2.6.31.5-127.fc12.x86_64
Samba: 3.4.2-47.fc12.x86_64
Greyhole: hda-greyhole-0.7.5-1.x86_64

2.
http://fpaste.org/DPSP/
http://fpaste.org/7KwS/

3.
/dev/mapper/VolGroup-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sdc1 on /boot type ext4 (rw)
/dev/sda1 on /var/hda/files/drives/drive1 type ext4 (rw)
/dev/sdb1 on /var/hda/files/drives/drive2 type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
gvfs-fuse-daemon on /home/Workdawg/.gvfs type fuse.gvfs-fuse-daemon (rw,nosuid,nodev,user=Workdawg)

Disk /dev/sda: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x0001190b

Device Boot Start End Blocks Id System
/dev/sda1 1 243202 1953513472 83 Linux

Disk /dev/sdb: 640.1 GB, 640135028736 bytes
255 heads, 63 sectors/track, 77825 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

Device Boot Start End Blocks Id System
/dev/sdb1 1 77825 625129281 83 Linux

Disk /dev/sdc: 20.4 GB, 20416757760 bytes
255 heads, 63 sectors/track, 2482 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x14844613

Device Boot Start End Blocks Id System
/dev/sdc1 * 1 26 204800 83 Linux
Partition 1 does not end on cylinder boundary.
/dev/sdc2 26 2482 19731833+ 8e Linux LVM

Disk /dev/dm-0: 16.0 GB, 15976103936 bytes
255 heads, 63 sectors/track, 1942 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

Disk /dev/dm-0 doesn't contain a valid partition table

Disk /dev/dm-1: 4227 MB, 4227858432 bytes
255 heads, 63 sectors/track, 514 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

Disk /dev/dm-1 doesn't contain a valid partition table
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
15G 3.6G 11G 26% /
tmpfs 1006M 560K 1005M 1% /dev/shm
/dev/sdc1 194M 23M 162M 13% /boot
/dev/sda1 1.8T 888G 853G 52% /var/hda/files/drives/drive1
/dev/sdb1 587G 37G 521G 7% /var/hda/files/drives/drive2

Greyhole Statistics
===================

Storage Pool
Total - Used = Free + Attic = Possible
/var/hda/files/drives/drive1/gh: 1834G - 888G = 853G + 8G = 860G
/var/hda/files/drives/drive2/gh: 587G - 37G = 520G + 0G = 520G

4.
1 | /var/hda/files/drivers/drive1 | 10 | 2011-02-12 12:42:49 | 2011-02-12 12:42:49
2 | /var/hda/files/drivers/drive2 | 10 | 2011-02-14 00:59:16 | 2011-02-14 00:59:16

5.
http://fpaste.org/oCnX/

6.
greyhole queue is 0 for all functions on all shares.

7. One of the missing files
http://fpaste.org/bwZg/

Another, that seems like it was there at first, then went missing: http://fpaste.org/KKBb/

[gboudreau]

Your problem is quite simple:
Greyhole has no idea it should look in /var/hda/files/drives/drive1/movies for files.
That directory is your old landing zone. And thus, is not in smb.conf anymore. /var/hda/files/drives/drive2/movies is in smb.conf
You'll want to manually move the files that are still in your old landing zones into your new ones.

The following commands will show you all the files you'll want to move into the new LZ:

Code: Select all

cd /var/hda/files/drives/drive1 find . -type f -a ! -path "./gh/*"

Move them manually into the appropriate folders in drive2 (/var/hda/files/drives/drive2/share_name, NOT inside the gh folder).

If you have too many to move manually, try this:

Code: Select all

cd /var/hda/files/drives/drive1 find . -type f -a ! -path "./gh/*" -exec mv "{}" "/var/hda/files/drives/drive2/{}"\;

That should move all the files in old LZ into the new LZ on drive2.

[Workdawg]

Thanks for the response. It looks like there's a small bug in your mass move script, but I'm not 100% sure what it is... and rather than try a couple things and make it worse, I'd prefer to ask up front.

the script you wrote up is missing a space before the \; at the end... I noticed that and added it.

After using cd to get to the right directory and using find to confirm all the files are there, I ran the script but it's telling me "cannot move ...... no such file or directory"
I noticed it looks like it's trying to copy files from (/var/hda/files/drives/drive1).movies/xxxxx to /var/hda/files/drives/drive2/./movies/xxxxx

Is the issue that it appears to be carrying the ./ over to the copy location and that's causing things to not work right? In my fairly limited experience, the ./ just indicates a relative file path, so I'm not sure if I should remove it or what.

Thanks!

[gboudreau]

A space was indeed missing before \;

Having ./ does nothing; you can leave it there.
The problem is probably that the target directories don't exists.

It might be easier to just copy, recursively, everything:

Code: Select all

cd cd /var/hda/files/drives/drive1 cp -r * /var/hda/files/drives/drive2/ greyhole --fsck

[Workdawg]

Ok, we are getting there, but I'm still having issues.

I used cp -r to copy everything back over, but it ended up filling up the 640gb drive I was using for the landing zone. greypool didn't start copying files off of the drive until after it filled up and the copy died. This recovered about half of the missing files to the share, but it made duplicate copies of them, so now I'm using up 1.4tb of space for about 900gb of actual data. I haven't done a final --fsck after all the copying happened because I let it run over night and I didn't have time to run it this morning, but that wouldn't help would it?

Should I just try manually copying all the missing files back over to the landing zone? What about the duplicate copies? Right now, I think I have 3 copies of the files that never went missing, two copies of the files that got recovered last night, and still only 1 copy of the files that are still missing. Is there any way to make greypool scan the pool and eliminate unwanted copies of files?

I'm starting to think it'd be easier to --going the 640gb drive and try and get all the files back on the two drives and start over, but my desktop machine is a win7 box, so I'd have to reformat the drive again... etc.

[gboudreau]

Run "greyhole --fsck"
It will take care of emptying the landing zone, and of keeping the correct number of file copies.

[Workdawg]

Okay, I thought it just managed the symlinks, but I'll give it a try when I get home tonight.

[Workdawg]

I ran a greyhole --fsck and it hasn't helped the situation.

Right now (numbered for easy reference below):
1. There is a copy all of the missing stuff (and the not missing stuff) in /var/hda/files/drives/drive1/movies (131 folders) - this appears to be a combo of links and files.
2. There is a copy of all the stuff I can see in the shares at /var/hda/files/drives/drive1/gh/movies (backup and music as well) (91 folders) - this appears to be full of actual files
3. There is a copy of all the stuff I can see in the shares at /var/hda/files/drives/drive2/movies (backup, music, etc) (91 folders) - this appears to be a bunch of links
4. There is a copy of some of the stuff also in /var/hda/files/drives/drive2/gh/movies (57 folders, this is the stuff that I was always able to see) -this is just a group of folders, no actual files... i tried to delete them but got permission denied

It looks like where there are links in 1, they are pointing to the files in 2. But not always, because I've found a copy of some files on in both locations. Though all the links I've found in 1 do go to 2. The links in 3 also point to the files in 2.

It seems like the easiest solution would be to blow away all the links, and create links to everything in 1... but the problem is that not all of the things in 1 are files, some are links to stuff in 2.

Ugh.

[gboudreau]

You should not have anything in /var/hda/files/drives/drive1/share_name
All files you have in those folders, you need to manually move into /var/hda/files/drives/drive2/share_name

Once that's done, use greyhole --fsck to have them moved into the pool (the gh subfolders).
From then on, /var/hda/files/drives/drive2/share_name will only contain links to either /var/hda/files/drives/drive1/gh/share_name or /var/hda/files/drives/drive2/gh/share_name, which are the storage pool directories.

[Workdawg]

Okay... so ./drives/drivex/gh is where files are actually stored, and .drives/drivex/(sharename) is where the links are... sweet.

Would those files be left over from when the landing zone was on that drive, or something like that? If I changed the landing zone to that location, and run and --fsck, then change it back, would that fix it?

I'll get to work manually doing this as doing it automatically has just seemed to screw things up worse.

Thanks!