[SOLVED] Grayhole deleted all data in a share

[jschlenk]

I installed Amahi and enabled Greyhole.

I think I may have lost a share (with my tv shows) while moving my data from a single hard drive (1TB) to a multi hard drive (1.5TB+500GB) greyhole configuration. I did a manual start with --fcsk, then while greypoole was copying my movies over I added some music through samba. I think this caused another instance of greypoole to launch causing a conflict with the first when it also attempted to copy the same movie (batman begins) that was already being copied. I have included what I think is the problem section from the greyhole.log file. The movie copy failed and afterward the space available went from 942 GB to 1.2 TB (bolded in the log file below), and all the links in my tv shows share were broken and the files gone from gh/TV Shows/. Luckily there wasn't any data lost that can't be retrieved (I have all the shows on DVD/Bluray), but this could have been a major disaster.

Code: Select all

Oct 4 20:54:52 7 write: Loading tombstones for Movies/Bluray/Batman Begins.mkv... Got 0 tombstones. Oct 4 20:54:52 7 write: Drives with available space: /var/hda/files/drives/drive2/gh (957GB avail) - /var/hda/files/drives/drive3/gh (425GB avail) Oct 4 20:54:52 7 write: Saving 1 tombstones for Movies/Bluray/Batman Begins.mkv Oct 4 20:54:52 7 write: Saving tombstones in /var/hda/files/drives/drive2/gh/.gh_graveyard/Movies/Bluray/Batman Begins.mkv Oct 4 20:54:52 7 write: Saving tombstones (backup) in /var/hda/files/drives/drive3/gh/.gh_graveyard_backup/Movies/Bluray/Batman Begins.mkv Oct 4 20:54:52 7 write: Copying file to /var/hda/files/drives/drive2/gh/Movies/Bluray/Batman Begins.mkv Oct 4 20:59:04 6 daemon: Greyhole (version 0.6.26) daemon started. Oct 4 20:59:04 7 daemon: Loading graveyard backup directories... Oct 4 20:59:04 7 daemon: Found 2 directories in the settings table. Oct 4 21:01:04 2 fsck: Can't insert fsck task: database is locked Oct 4 21:04:41 7 simplify_tasks: Simplifying pending tasks. Oct 4 21:04:41 7 simplify_tasks: Removing 3 duplicate pending tasks for Music/.DS_Store Oct 4 21:04:41 7 simplify_tasks: Removing 2 duplicate pending tasks for Music/U2/All That You Can't Leave Behind/._01 Beautiful Day.m4a Oct 4 21:04:41 7 simplify_tasks: Removing 2 duplicate pending tasks for Music/U2/All That You Can't Leave Behind/._02 Stuck In A Moment You Can't Get Out Of.m4a Oct 4 21:04:41 7 simplify_tasks: Removing 2 duplicate pending tasks for Music/U2/All That You Can't Leave Behind/._03 Elevation.m4a Oct 4 21:04:42 7 simplify_tasks: Removing 2 duplicate pending tasks for Music/U2/All That You Can't Leave Behind/._04 Walk On.m4a Oct 4 21:04:42 7 simplify_tasks: Removing 2 duplicate pending tasks for Music/U2/All That You Can't Leave Behind/._05 Kite.m4a Oct 4 21:04:42 7 simplify_tasks: Removing 2 duplicate pending tasks for Music/U2/All That You Can't Leave Behind/._06 In A Little While.m4a Oct 4 21:04:42 7 simplify_tasks: Removing 2 duplicate pending tasks for Music/U2/All That You Can't Leave Behind/._07 Wild Honey.m4a Oct 4 21:04:42 7 simplify_tasks: Removing 2 duplicate pending tasks for Music/U2/All That You Can't Leave Behind/._08 Peace On Earth.m4a Oct 4 21:04:42 7 simplify_tasks: Removing 2 duplicate pending tasks for Music/U2/All That You Can't Leave Behind/._09 When I Look At The World.m4a Oct 4 21:04:44 7 simplify_tasks: Removing 2 duplicate pending tasks for Music/U2/All That You Can't Leave Behind/._10 New York.m4a Oct 4 21:04:44 7 simplify_tasks: Removing 2 duplicate pending tasks for Music/U2/All That You Can't Leave Behind/._11 Grace.m4a Oct 4 21:04:44 7 simplify_tasks: Removing 1 duplicate pending tasks for Music/U2/All That You Can't Leave Behind/01 Beautiful Day.m4a Oct 4 21:04:44 7 simplify_tasks: Removing 1 duplicate pending tasks for Music/U2/All That You Can't Leave Behind/02 Stuck In A Moment You Can't Get Out Of.m4a Oct 4 21:04:44 7 simplify_tasks: Removing 1 duplicate pending tasks for Music/U2/All That You Can't Leave Behind/03 Elevation.m4a Oct 4 21:04:44 7 simplify_tasks: Removing 1 duplicate pending tasks for Music/U2/All That You Can't Leave Behind/04 Walk On.m4a ......Oct 4 21:17:58 7 write: Saving 1 tombstones for Movies/Bluray/Batman Begins.mkv Oct 4 21:17:58 7 write: Saving tombstones in /var/hda/files/drives/drive2/gh/.gh_graveyard/Movies/Bluray/Batman Begins.mkv Oct 4 21:17:58 7 write: Saving tombstones (backup) in /var/hda/files/drives/drive3/gh/.gh_graveyard_backup/Movies/Bluray/Batman Begins.mkv

[gboudreau]

When Greyhole remove files, it says so in the log file. Nothing appears in your log file between the two bold lines, which indicates Greyhole didn't delete those files.

What's more, Greyhole doesn't really delete files with the default config Amahi uses; all files that are 'deleted' are in fact moved into the 'attic' (the equivalent of a recycle bin), so the space freed by deleting files isn't really available until you manually empty the attic.

And finally, Greyhole isn't multi-threaded, so if the log says it's working on copying a MKV file, it's not doing anything else at the same time.

My guess is that another process removed those files, or you made a manipulation error. You can try to see if you did by using the 'history' command, to try and see what you might have done on the command line, if you did use the command line. If you used the Amahi desktop to move files around, there's no trace really, but maybe you could have mixed up the source folder with another window that was open to .../gh/TV Shows/ ?

The possibilities are pretty much endless, but I'm positive it's not Greyhole that removed those file copies.

[jschlenk]

I wasn't using the machine for anything else when this occurred. At the most I was just browsing around in nautilus, checking on greyhole's progress. I noticed the problem when I saw that my free hdd space had increased.

The only data I lost was the data copied over during the current greyhole copy session, when I added my TV Show and Movies shares to the poole and started greyhole. I had previously added other shares to the greyhole pool, and those weren't deleted. I agree it seems weird that greyhole would delete files without logging it, but I don't know what else could have happened. Maybe something in samba?

I tried to browse to the root of both the pool drives in nautilus and and got message like "this folder could not be opened, maybe it has been recently deleted". After a couple tries it worked again but I found all the files and directories in ../gh/TV Shows were gone.

Also the music files weren't moved into the storage pool. In the log it has a bunch of "Removed X duplicate pending tasks" for the music.

[gboudreau]

Maybe it has to do with where your shares are, what drives are in your pool, etc.

I'd like to see your config:

Code: Select all

yum -y install fpaste fpaste /etc/greyhole.conf fpaste /etc/samba/smb.conf mount | fpaste

[jschlenk]

greyhole.conf:
http://fpaste.org/qjMI/

smbd.conf:
http://fpaste.org/uLo9/

mount:
http://fpaste.org/rjzJ/

[gboudreau]

Your greyhole.conf has no storage pool lines.
I doubt Greyhole likes that (or even runs).
Maybe you removed partitions from your storage pool from the Amahi dashboard?

If you go back in that page, is any partition selected?
If you unselected them yourself, which ones were selected when this happened?
Did you change any other greyhole/shares configuration since then?

[gboudreau]

If you could give me remote SSH access, I'd like to have a look and see what I can find there.

[jschlenk]

I removed all the hard drives from the storage pool with the amahi dashboard afterwards to stop anything else from happening.

I removed sdc1 (1.5TB internal) and sdb1 (500GB usb drive).

I have made no further changes since then. I just ran extundelete to recovery whatever files I could.

I will setup an account for you and PM you the login details

[gboudreau]

You know what I think happened?
Your drive2 (sdc1) was not mounted when greyhole moved file copies from /var/hda/files/videos/tv shows into /var/hda/files/drives/drive2/gh/tv shows.
Since drive2 wasn't really mounted, the file copies ended up on your sda6 drive (mounted as /var/hda/files).
When later drive2 was mounted, the file copies were hidden behind the mount!

Try this: umount /var/hda/files/drives/drive2
Then look in /var/hda/files/drives/drive2/gh/tv shows
I'm pretty sure your files are all there.

There is already a bug file in the Amahi bug tracker to prevent this from happening.
The problem is that Amahi allowed GH to use the /var/hda/files/drives/drive2/gh folder even when it wasn't a mount point.
If the situation I described above is what happened to you, I invite you to weight in in the bug report I created about this a while ago: http://bugs.amahi.org/issues/show/611

[jschlenk]

I tried unmounding /dev/sdc1 but the ../drives/drive2 folder is now empty. I think the files are lost for good because sda6 only had about 16 GB left of free space before, and now it has 300.