Greyhole messed up (again) need help

[flickeringsight]

So, today I noticed that a few dozen files went missing since last night in my movies folder. I dug around the /gh/ directories and, thankfully found the files. What seems to have happened is that I changed a few folder names on the Movies share (it was the share and not the actual directory on the server) and Greyhole didn't keep up with it.

The nature of the changes were like this: /movies/Show/Show Season 1/ to /movies/Show/Season 1/

In the /gh/ directories the subfolders are still named /Show Season 1/.

So I ran greyhole -f and that didn't help. Then I ran greyhole --debug /var/hda/files/movies/Show it listed a bunch or 'write pending task' for files in the one subdirectory I didn't change, and hadn't lost any files, followed by this gem:

Code: Select all

From filesystem =============== Landing Zone: ls: cannot access /hda/files/movies/Show: No such file or directory

So what do I do now?


P.S. One other nagging issue. Whenever I issue a command to Greyhole, it spits this out:

Code: Select all

PHP Warning: date_default_timezone_get(): It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PST/-8.0/no DST' instead in /usr/bin/greyhole on line 39

P.P.S. I also just noticed that in the "Servers" section of the Dashboard it shows that Grayhole Server as being off, even though it does respond to commands. I tried giving it a service greyhole restart and it responded with this.

Code: Select all

PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 4686721024 bytes) in /usr/bin/greyhole on line 1916

So, apparently it's trying to allocate 4.3 gigabytes of memory?

Here's what the log has to say when I try and start the grehole service.

Code: Select all

Dec 15 07:46:34 7 daemon: Loading graveyard backup directories... Dec 15 07:46:34 7 daemon: Found 2 directories in the settings table. Dec 15 07:46:34 7 daemon: More than 900 tasks queued... Won't queue any more at this time. Dec 15 07:46:34 7 simplify_tasks: Simplifying pending tasks. Dec 15 07:46:35 7 simplify_tasks: More than 900 tasks queued... Won't queue any more at this time. Dec 15 07:46:35 7 fsck_file: Now working on task ID 920813: fsck_file Music/Artist/album/file.mkv Dec 15 07:46:35 7 fsck_file: Music/Artist/album/file.mkv...

It's at that last log entry that greyhole take a dump. Oddly, the file it's trying to fsck (the name I've changed to protect the innocent ) isn't supposed to be in the music directory, it's a movie file located in the /Movies share. I don't know why it's listed as being there - perhaps corruption of the greyhole DB?

I've looked in the graveyard directory where it's getting stuck and cannot find tombstone for the mkv file it's looking for.

Code: Select all

ls '/var/hda/files/gh/.gh_graveyard/Music/Artist/album/'

I have made a little progress. In an effort to rebuild the /Movies share I fist backed up the contents of the landing zone (entirely symlinks) and then blew them all out with

Code: Select all

rm -rf /var/hd/files/movies/*

I then scheduled a fsck task on the /Music directory, telling it not to walk the graveyard, since that's where the problem is occurring, so maybe it can get past the issue.

Code: Select all

greyhole -fyd /var/hda/files/music

Then manually launched the daemon

Code: Select all

greyhole -D

It seems to be working! Hopefully it'll rebuild the /Movies share. However, I have concerns about the root issue of the /Music tombstone it's looking for, and getting stuck on.

Failure. So greyhole worked for a few minutes fsck-ing when it took a dump at the same spot it always does,

Last Log Entry

Code: Select all

task ID 920813: fsck_file Music/Artist/album/file.mkv Loading tombstones for Music/Artist/album/file.mkv...

Terminal Return

Code: Select all

PHP Fatal error: Allowed memory size of 167772160 bytes exhausted (tried to allocate 4686721024 bytes) in /usr/bin/greyhole on line 1919

[flickeringsight]

Holy crap, I think I fixed it. After digging around posts with similar problems and taking a shot in the dark, I issued this command to try and clear out the offending queued task:

Code: Select all

sqlite3 /var/cache/greyhole.sqlite 'delete from tasks where ID = 920813'

And then tried to restart the greyhole service:

Code: Select all

service greyhole condrestart

It launched successfully! I'll keep an eye on the logs...

[flickeringsight]

OK, Greyhole seems to have finished fsck-ing after around 24 hours of work, much of which seemed to be replicated effort on the files and directories. It ran into the same (I have to assume) corrupt tombstone path, which caused the initial problem, about four times, which crashed greyhole. I used the same solution as above to get past the problem until it finished all tasks and is now sleeping.

[flickeringsight]

So, the problem persists. Once every day or two Greyhole hangs while attempting to fsck a file that does not exist.

Code: Select all

Dec 19 08:28:09 7 fsck_file: Now working on task ID 1276153: fsck_file Music/Alanis Morissette/jagged little pill/inf-raven720p.mkv Dec 19 08:28:09 7 fsck_file: Loading tombstones for Music/Alanis Morissette/jagged little pill/inf-raven720p.mkv...

*HANG*

Each time a can alleviate the problem by issuing...

Code: Select all

sqlite3 /var/cache/greyhole.sqlite 'delete from tasks where ID = 1276153'

...but it keeps turning up. How do I fix this permanently???

[bigfoot65]

Is this Fedora 14 or Ubuntu? I would recommend you convert Greyhole to MySQL. There is guidance in the wiki.

Also have you considered upgrading Greyhole to the latest version? It might fix your issue. The version integrated with Fedora 14 Amahi is outdated and the newest version adds features/fixes some bugs.