Backups Disappearing?

[bbuzzard]

Hi, with help from the group, I've gotten the greyhole server to stay running after a backup of my Windows machines. However, I'm noticing that the backups that are created are disappearing after 24 hours or so.

I have 2 Windows machines on my network (1 x XP, 1 x W7). When I run TrueImage and back up an image of the drives to shares I created for each (for example /var/hda/files/drives/sdb1/Quark_Backup for one, same place, different share name for the other) everything goes fine. I even created a rescue CD ISO and put it in each share so I could make recovery CDs later. The backups are named something like "Quark_Backup_06_30_10.tib" or something like that.

I backed up each machine on Tuesday Night. When I checked it today (Thursday morning), the backups were gone but the ISO files were still there in each share.

Should I be creating the backup shares somewhere else, or am I doing this whole thing wrong? When I get home tonight I'll try to take a look at the greyhole log to see if I can tell when the files were removed.

[gboudreau]

Indeed, check greyhole.log, or:

Code: Select all

greyhole --debug Quark_Backup_06_30_10.tib

to see what happened recently with Quark_Backup_06_30_10.tib

[cpg]

This does not seem greyhole related (from what the OP said in the post).

Try troubleshooting the shares and see if that helps. Some versions of windows some times make it look like some files disappeared.

In any case, this is something we care about - reliability of backups, so let us know how it went.

[bbuzzard]

Just got home. After I put my boy in bed I'm going to dig into it. The strange thing is, there were 2 files in each share (1 called XXX_Backup_MM_YY_YY.tib, and 1 called AcronisMedia.iso), but only the backup disappeared in each share, leaving the .iso behind.

I'm no Linux expert, so I might just have the shares wrong. But I have a ton of photos in another share that are still there.

[bbuzzard]

OK, I did the debug thing. The long output is below.

It may also be of interest that both of the backups (one for each machine) had the same name but were in different shares, not what I intended. Both the shares use the pool, but have no extra copies specified.

The debug output:

[root@AMAHI ~]# greyhole --debug MyBackup__6_28_2010.tib
Debugging file operations for file named "/MyBackup__6_28_2010.tib"

From DB
=======

From logs
=========
Jun 28 23:59:10 7 unlink: Removing tombstones for Quark_Backup/C_VOL-b005.spf_{A7EEFD72-20B8-422C-847F-A5EB21F6BF05}.tmp
Jun 28 23:59:10 7 rename: Now working on task ID 54272: rename Quark_Backup/MyBackup__6_28_20101.tib -> Quark_Backup/MyBackup__6_28_2010.tib
Jun 28 23:59:10 6 rename: File renamed: /var/hda/files/drives/sdb1/Quark_Backup/MyBackup__6_28_20101.tib -> /var/hda/files/drives/sdb1/Quark_Backup/MyBackup__6_28_2010.tib
Jun 28 23:59:11 7 rename: Loading tombstones for Quark_Backup/MyBackup__6_28_20101.tib... Got 0 tombstones.
Jun 28 23:59:11 7 rename: Loading tombstones for Quark_Backup/MyBackup__6_28_2010.tib... Got 0 tombstones.
Jun 28 23:59:11 7 rename: Loading tombstones for Quark_Backup/MyBackup__6_28_20101.tib... Got 0 tombstones.
Jun 28 23:59:11 6 rename: File created: Quark_Backup/MyBackup__6_28_2010.tibPHP Warning [2]: filesize(): stat failed for /var/hda/files/drives/sdb1/Quark_Backup/MyBackup__6_28_2010.tib in /usr/bin/greyhole on line 1329
Jun 28 23:59:11 7 rename: - 0.00B
Jun 28 23:59:11 7 rename: Loading tombstones for Quark_Backup/MyBackup__6_28_2010.tib... Got 0 tombstones.
Jun 28 23:59:12 7 rename: Drives with available space: /var/hda/files/drives/sdb1/gh (241GB avail) - /var/hda/files/drives/sdc1/gh (137GB avail) - /var/hda/files/drives/sdd1/gh (137GB avail)
Jun 28 23:59:12 7 rename: Saving 1 tombstones for Quark_Backup/MyBackup__6_28_2010.tib
Jun 28 23:59:12 7 rename: Saving tombstones in /var/hda/files/drives/sdb1/gh/.gh_graveyard/Quark_Backup//MyBackup__6_28_2010.tib
Jun 28 23:59:12 7 rename: Copying file to /var/hda/files/drives/sdb1/gh/Quark_Backup/MyBackup__6_28_2010.tib
Jun 28 23:59:12 4 rename: PHP Warning [2]: fileowner(): stat failed for /var/hda/files/drives/sdb1/Quark_Backup/MyBackup__6_28_2010.tib in /usr/bin/greyhole on line 2885
Jun 28 23:59:12 4 rename: PHP Warning [2]: filegroup(): stat failed for /var/hda/files/drives/sdb1/Quark_Backup/MyBackup__6_28_2010.tib in /usr/bin/greyhole on line 2886
Jun 28 23:59:12 4 rename: PHP Warning [2]: fileperms(): stat failed for /var/hda/files/drives/sdb1/Quark_Backup/MyBackup__6_28_2010.tib in /usr/bin/greyhole on line 2887
Jun 28 23:59:12 4 rename: Failed file copy. Skipping.
Jun 28 23:59:12 7 rename: Saving 1 tombstones for Quark_Backup/MyBackup__6_28_2010.tib
Jun 28 23:59:12 7 rename: Saving tombstones in /var/hda/files/drives/sdb1/gh/.gh_graveyard/Quark_Backup//MyBackup__6_28_2010.tib
Jun 28 23:59:12 7 unlink: Now working on task ID 54275: unlink Quark_Backup/AcronisMedia.iso
Jun 29 22:40:57 7 fsck: Loading tombstones for Quark_Backup/AcronisMedia.iso... Got 1 tombstones.
Jun 29 22:40:57 6 fsck: /var/hda/files/drives/sdb1/Quark_Backup/MyBackup__6_28_2010.tib is missing from the share directory. Will try to re-create it, if copies are available.
Jun 29 22:40:57 7 fsck: Loading tombstones for Quark_Backup/MyBackup__6_28_2010.tib... Got 1 tombstones.
Jun 29 22:40:57 4 fsck: WARNING! No copies of this file are available. Deleting from share.
Jun 29 22:41:10 7 fsck: Deleted copy at /var/hda/files/drives/sdb1/Quark_Backup//MyBackup__6_28_2010.tib
Jun 29 22:41:10 7 fsck: Saving 1 tombstones for Quark_Backup/MyBackup__6_28_2010.tib
Jun 29 22:41:10 7 fsck: Saving tombstones in /var/hda/files/drives/sdb1/gh/.gh_graveyard/Quark_Backup//MyBackup__6_28_2010.tib
Jun 29 22:41:10 7 fsck: Entering /var/hda/files/drives/sdb1/Sun_Backup
--
Jun 29 22:41:30 7 fsck: Saving tombstones in /var/hda/files/drives/sdb1/gh/.gh_graveyard/Quark_Backup//C_VOL-b002.spf
Jun 29 22:41:30 6 fsck: /var/hda/files/drives/sdb1/Quark_Backup//MyBackup__6_28_2010.tib is missing from the share directory. Will try to re-create it, if copies are available.
Jun 29 22:41:30 7 fsck: Loading tombstones for Quark_Backup/MyBackup__6_28_2010.tib... Got 1 tombstones.
Jun 29 22:41:30 4 fsck: WARNING! No copies of this file are available. Deleting from share.
Jun 29 22:41:30 7 fsck: Saving 1 tombstones for Quark_Backup/MyBackup__6_28_2010.tib
Jun 29 22:41:30 7 fsck: Saving tombstones in /var/hda/files/drives/sdb1/gh/.gh_graveyard/Quark_Backup//MyBackup__6_28_2010.tib
Jun 29 22:41:30 7 fsck: Entering graveyard /var/hda/files/drives/sdb1/gh/.gh_graveyard/Sun_Backup
Jun 30 17:56:53 7 fsck: Saving tombstones in /var/hda/files/drives/sdb1/gh/.gh_graveyard/Quark_Backup//C_VOL-b002.spf
Jun 30 17:56:53 6 fsck: /var/hda/files/drives/sdb1/Quark_Backup//MyBackup__6_28_2010.tib is missing from the share directory. Will try to re-create it, if copies are available.
Jun 30 17:56:53 7 fsck: Loading tombstones for Quark_Backup/MyBackup__6_28_2010.tib... Got 1 tombstones.
Jun 30 17:56:53 4 fsck: WARNING! No copies of this file are available. Deleting from share.
Jun 30 17:56:53 7 fsck: Saving 1 tombstones for Quark_Backup/MyBackup__6_28_2010.tib
Jun 30 17:56:53 7 fsck: Saving tombstones in /var/hda/files/drives/sdb1/gh/.gh_graveyard/Quark_Backup//MyBackup__6_28_2010.tib
Jun 30 17:56:53 7 fsck: Entering graveyard /var/hda/files/drives/sdb1/gh/.gh_graveyard/Sun_Backup
--
Jul 1 00:01:49 7 fsck: Saving tombstones in /var/hda/files/drives/sdb1/gh/.gh_graveyard/Quark_Backup//C_VOL-b002.spf
Jul 1 00:01:49 6 fsck: /var/hda/files/drives/sdb1/Quark_Backup//MyBackup__6_28_2010.tib is missing from the share directory. Will try to re-create it, if copies are available.
Jul 1 00:01:49 7 fsck: Loading tombstones for Quark_Backup/MyBackup__6_28_2010.tib... Got 1 tombstones.
Jul 1 00:01:49 4 fsck: WARNING! No copies of this file are available. Deleting from share.
Jul 1 00:01:49 7 fsck: Saving 1 tombstones for Quark_Backup/MyBackup__6_28_2010.tib
Jul 1 00:01:49 7 fsck: Saving tombstones in /var/hda/files/drives/sdb1/gh/.gh_graveyard/Quark_Backup//MyBackup__6_28_2010.tib
Jul 1 00:01:49 7 fsck: Entering graveyard /var/hda/files/drives/sdb1/gh/.gh_graveyard/Sun_Backup
Jul 1 07:42:34 7 fsck: Saving tombstones in /var/hda/files/drives/sdb1/gh/.gh_graveyard/Quark_Backup//C_VOL-b002.spf
Jul 1 07:42:34 6 fsck: /var/hda/files/drives/sdb1/Quark_Backup//MyBackup__6_28_2010.tib is missing from the share directory. Will try to re-create it, if copies are available.
Jul 1 07:42:34 7 fsck: Loading tombstones for Quark_Backup/MyBackup__6_28_2010.tib... Got 1 tombstones.
Jul 1 07:42:34 4 fsck: WARNING! No copies of this file are available. Deleting from share.
Jul 1 07:42:34 7 fsck: Saving 1 tombstones for Quark_Backup/MyBackup__6_28_2010.tib
Jul 1 07:42:34 7 fsck: Saving tombstones in /var/hda/files/drives/sdb1/gh/.gh_graveyard/Quark_Backup//MyBackup__6_28_2010.tib
Jul 1 07:42:34 7 fsck: Entering graveyard /var/hda/files/drives/sdb1/gh/.gh_graveyard/Sun_Backup
--
Jul 2 00:01:15 7 fsck: Saving tombstones in /var/hda/files/drives/sdb1/gh/.gh_graveyard/Quark_Backup//C_VOL-b002.spf
Jul 2 00:01:15 6 fsck: /var/hda/files/drives/sdb1/Quark_Backup//MyBackup__6_28_2010.tib is missing from the share directory. Will try to re-create it, if copies are available.
Jul 2 00:01:15 7 fsck: Loading tombstones for Quark_Backup/MyBackup__6_28_2010.tib... Got 1 tombstones.
Jul 2 00:01:15 4 fsck: WARNING! No copies of this file are available. Deleting from share.
Jul 2 00:01:15 7 fsck: Saving 1 tombstones for Quark_Backup/MyBackup__6_28_2010.tib
Jul 2 00:01:15 7 fsck: Saving tombstones in /var/hda/files/drives/sdb1/gh/.gh_graveyard/Quark_Backup//MyBackup__6_28_2010.tib
Jul 2 00:01:15 7 fsck: Entering graveyard /var/hda/files/drives/sdb1/gh/.gh_graveyard/Sun_Backup

From filesystem
===============
Landing Zone:
ls: cannot access /MyBackup__6_28_2010.tib: No such file or directory

Graveyard:

File copies:
[root@AMAHI ~]#

[bbuzzard]

OK, one last update before I'm done for the night.

I decided to run a backup on just one machine and see what greyhole said about it.

This is what I got:

Jul 2 23:27:11 6 daemon: Greyhole (version 0.6.12) daemon started.
Jul 2 23:27:11 7 simplify_tasks: Simplifying pending tasks.
Jul 2 23:27:11 7 write: Now working on task ID 0: write Quark_Backup//quarkbackup__7_2_2010.tib
Jul 2 23:27:12 6 write: File created: Quark_Backup//quarkbackup__7_2_2010.tibPHP Warning [2]: filesize(): stat failed for /var/hda/files/drives/sdb1/Quark_Backup/quarkbackup__7_2_2010.tib in /usr/bin/greyhole on line 1338
Jul 2 23:27:12 7 write: - 0.00B
Jul 2 23:27:12 7 write: Loading tombstones for Quark_Backup/quarkbackup__7_2_2010.tib... Found a tombstone, pointing to '/var/hda/files/drives/sdb1/gh/Quark_Backup/quarkbackup__7_2_2010.tib', with state = 'Gone'. We just want 'OK' tombstones; will not use this tombstone.
Jul 2 23:27:12 7 write: Got 0 tombstones.
Jul 2 23:27:12 7 write: Drives with available space: /var/hda/files/drives/sdb1/gh (249GB avail) - /var/hda/files/drives/sdc1/gh (137GB avail) - /var/hda/files/drives/sdd1/gh (137GB avail)
Jul 2 23:27:12 7 write: Saving 1 tombstones for Quark_Backup/quarkbackup__7_2_2010.tib
Jul 2 23:27:12 7 write: Saving tombstones in /var/hda/files/drives/sdb1/gh/.gh_graveyard/Quark_Backup//quarkbackup__7_2_2010.tib
Jul 2 23:27:12 7 write: Copying file to /var/hda/files/drives/sdb1/gh/Quark_Backup/quarkbackup__7_2_2010.tib
Jul 2 23:27:12 4 write: PHP Warning [2]: fileowner(): stat failed for /var/hda/files/drives/sdb1/Quark_Backup//quarkbackup__7_2_2010.tib in /usr/bin/greyhole on line 2978
Jul 2 23:27:12 4 write: PHP Warning [2]: filegroup(): stat failed for /var/hda/files/drives/sdb1/Quark_Backup//quarkbackup__7_2_2010.tib in /usr/bin/greyhole on line 2979
Jul 2 23:27:12 4 write: PHP Warning [2]: fileperms(): stat failed for /var/hda/files/drives/sdb1/Quark_Backup//quarkbackup__7_2_2010.tib in /usr/bin/greyhole on line 2980
Jul 2 23:27:12 4 write: Failed file copy. Will mark this tombstone 'Gone', and re-process this write task.
Jul 2 23:27:12 7 write: Saving 1 tombstones for Quark_Backup/quarkbackup__7_2_2010.tib
Jul 2 23:27:12 7 write: Saving tombstones in /var/hda/files/drives/sdb1/gh/.gh_graveyard/Quark_Backup//quarkbackup__7_2_2010.tib
Jul 2 23:27:12 2 write: Can't insert write task: PRIMARY KEY must be unique

[gboudreau]

Jul 2 23:27:12 6 write: File created: Quark_Backup//quarkbackup__7_2_2010.tib
PHP Warning [2]: filesize(): stat failed for /var/hda/files/drives/sdb1/Quark_Backup/quarkbackup__7_2_2010.tib in /usr/bin/greyhole on line 1338
Jul 2 23:27:12 7 write: - 0.00B

Jul 2 23:27:12 7 write: Copying file to /var/hda/files/drives/sdb1/gh/Quark_Backup/quarkbackup__7_2_2010.tib
Jul 2 23:27:12 4 write: PHP Warning [2]: fileowner(): stat failed for /var/hda/files/drives/sdb1/Quark_Backup//quarkbackup__7_2_2010.tib in /usr/bin/greyhole on line 2978
Jul 2 23:27:12 4 write: PHP Warning [2]: filegroup(): stat failed for /var/hda/files/drives/sdb1/Quark_Backup//quarkbackup__7_2_2010.tib in /usr/bin/greyhole on line 2979
Jul 2 23:27:12 4 write: PHP Warning [2]: fileperms(): stat failed for /var/hda/files/drives/sdb1/Quark_Backup//quarkbackup__7_2_2010.tib in /usr/bin/greyhole on line 2980
Jul 2 23:27:12 4 write: Failed file copy. Will mark this tombstone 'Gone', and re-process this write task.

Those are weird... It's as if the .tib file is unreadable at this time...
At 23:27:11, was the backup still being created? i.e. was Greyhole trying to work on a file that was still being written to?
Also, what file system is your /var/hda/files/drives/sdb1 drive formated in? (NTFS, Linux ext3/4?)
You can check by executing this command:

Code: Select all

mount

(Paste the result in your reply here.)

Jul 2 23:27:12 2 write: Can't insert write task: PRIMARY KEY must be unique

This is a known issue with SQLite. You can apply the fix found here: http://bugs.amahi.org/issues/show/516#note-1
See the part that says: "To fix, execute those commands, as root [...]"

[bbuzzard]

The result of the mount command:

root@AMAHI ~]# mount
/dev/sda3 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sda1 on /boot type ext4 (rw)
/dev/sdb1 on /var/hda/files/drives/sdb1 type ext4 (rw)
/dev/sdc1 on /var/hda/files/drives/sdc1 type ext4 (rw)
/dev/sdd1 on /var/hda/files/drives/sdd1 type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
gvfs-fuse-daemon on /home/bbuzzard/.gvfs type fuse.gvfs-fuse-daemon (rw,nosuid,nodev,user=bbuzzard)

it's ext4.

The backup was completed at 23:11. If it's any help, the backup is 49 Gig. After it finished, I began looking at the log. The latest backup is still there, but maybe it's because the server is stopped.

As for bug fix 516, I had already done that to fix a different issue (backups stopping server). I did it again, and the same issue (primary key must be unique) is still there.

Thanks for all your attention on this issue.

[gboudreau]

Can't insert write task: PRIMARY KEY must be unique

This part is indeed not related to bug #516. It happens when the task with ID 0 fails. Task with ID 0 is a retry on a failed task, so when the retry fails too, it tries to insert another task with ID 0, which gives the above error, and kills Greyhole.
I'll fix this issue in the upcoming GH release.

To fix this for now, execute this:

Code: Select all

sqlite3 /var/cache/greyhole.sqlite 'delete from tasks where id = 0'; service greyhole restart tail -f /var/log/greyhole.log

Wait until the greyhole.log starts saying "Nothing to do... Sleeping."
Then, delete all previous .tib files from your shares, and wait again for Greyhole to start sleeping.
At that point, Ctrl-C the running "tail -f".

Then, execute the following (as one command):

Code: Select all

service greyhole stop; mv /var/log/greyhole.log /var/log/greyhole.log.bak; service greyhole start

At this point, start a backup, and wait for it to complete.
Once that's complete, and Greyhole seems to be done doing whatever it tried to do, fpaste the greyhole.log:

Code: Select all

cat /var/log/greyhole.log | fpaste

Post the URL you receive in a new post here.

[bbuzzard]

Attached is the log from the backup:

http://fpaste.org/XxOf/

It might not mean anything, but as soon as I used the code you provided to fix my greyhole loop, the backup tib file was deleted from the share.