Amahi Forums

This follows up another thread I posted about a week or so back regarding greyhole.log spinning up too large such that I would run of space on my 20g system partition in just hours.. the greyhole.log file would get up to 8g inside an hour.. I'd delete it and it would rebuild...

This occurred after I move a large amount of files from a backed up system drive from another machine into my pool and also installed a new 2Tb drive into the system. The system is composed of 9 total drives and about 8Tb.

I solve the greyhole.log problem by created a cronjob to rotate the log initially every 15 minutes and then tuned it down to every half hour. This seemed to solve the problem. I now have plenty of free gb on the partition.

I'm now experiencing other issues. The first symptoms was the inability to access shares... the next symptoms was the inability to bring up the hda page...
I checked my /var/log/greyhole.log and there was no log... service was running.. I restarted it.. not solved.. I rebooted and the greyhole.log log started generating... but the issues persisted...

I tried: but get that error...

Also.. tail /var/log/messages: I did some googling and found the issue might be related to running out of inodes: So.. you can see I"m out of inodes on the system partition... I strongly suspect it is the greyhole spool file, but I'm not 100% sure...
Question: Can I kill those spooled files, delete unnecessary files in my shares I know I don't need that were copied over recently (e.g., /documents and settings/application data/"gobs and gobs of crap in those directories I don't need that got copied over from decommissioned hard drive"...).. and then re-run an fsck to let the spool rebuild??

Or will I blow things up really good if I do that??

Knowledgeable input would be immensely appreciated!

jbmia

You could kill the spooled files, but there are a few caveats to that. Greyhole will lose any knowledge of file operations that were performed and are represented in those spool files. (i.e. any renames, moves, deletes, writes, etc) Most of those don't carry much risk, if you moved or renamed something those changes will just be undone upon the next fsck since Greyhole will not have transferred those operations to the pool drives. If, however, you've got any shares with 2x (or more) copies and there are write operations in the spool, your files could get into an inconsistent state. (i.e. one copy is updated and the others are not.)

I am curious, though, how it managed to create such a prolific number of spool files without consuming them. Do you have a large number of pending tasks? Are the tasks actually being processed?

Thanks for the reply! I'm going to go ahead and delete the files...

You know, I thought that the spool files were the pending tasks... so, I'm not sure the answer there. Am I misunderstanding the nature of the spooled files??

But, to try to answer your question as to whether greyhole is consuming the spooled tasks (which again is what I assume they are) I'm not sure 100% either way... On the one hand, per below, the spooled count is not descreasing... its actually increased by a few files since my intial post: The box is headless.. and is running in text mode (no X) in order to save memory and so on... with that, I can see the system just sitting there flying through messages in the terminal window.. many of which start with a greyhole related message (e.g., ... found a tombstone..., etc), but I'm not sure what it's actually doing cause they're flying so quickly across the screen.

Again, /var/log/messages: So, if it needed some tmp file space on the system partition, to process those spooled tasks, I would imagine they'd be erroring out due to "no space left on device"...

Any other way to tell if it's consuming the spool?

The greyhole.log,, logrotate, and compress process seems to be barfing on the lack of space as well.. if you let it run, it'll create a greyhole.log file, but once the half hourly logrotate process is invoked, it throws up on the compress and save portions cause of the disk space I think: So, once that file is deleted, greyhole.log is restarted since we have some space again, and: So, if greyhole couldn't write to it's log, what would it do? Continue processing spooled files? or other tasks? Or, would it stop actually consuming tasks? Does greyhole need tmp file space on the system partition that would make it throw up that partitions maxed out inode state? Interesting questions I don't have the answer for.... Let me know what you think...

Thanks again for your assistance! Much appreciation!

jbmia

You know, I thought that the spool files were the pending tasks... so, I'm not sure the answer there. Am I misunderstanding the nature of the spooled files??

There's a difference between pending tasks and the files in the spool directory. Once Greyhole reads a spool file and loads that task into the database it deletes the spool file. So what you're seeing in that directory are tasks Greyhole hasn't even gotten to yet. In general Greyhole keeps about a thousand tasks loaded in the database at any given time, so if your spool files aren't getting consumed, it sounds like tasks aren't being processed.

Again, /var/log/messages:

Code: Select all

Jul 21 04:04:25 9745 nmbd[31295]: wins_write_database: Can't open /var/lib/samba/wins.dat.31295. Error was No space left on device[/quote] I'm assuming the above is the VFS plugin for Samba that creates the spool files for Greyhole crapping out since you're out of space (or inodes). [quote]So, if greyhole couldn't write to it's log, what would it do? Continue processing spooled files? or other tasks? Or, would it stop actually consuming tasks? Does greyhole need tmp file space on the system partition that would make it throw up that partitions maxed out inode state? Interesting questions I don't have the answer for.... Let me know what you think...[/quote] I think the better question is whether Greyhole would crash if it can't write to its logs. I'm not sure if it will, I haven't run that scenario before. You can verify when/if you hit this again, though. As root you can run `service greyhole status` if it's not running, then obviously it won't be spooling / processing tasks. If it continues running when logs cannot be written to and your landing zone is on the full partition I see further complications arising. More to the point, you may just want to remove all the tombstones and files manually for the files you want to get rid of since it seems like you're putting a heavy load on Greyhole. You'd have to remove the folder from both the graveyards and the share paths on the pool drives. You could also remove them from the landing zone but an fsck will remove the symlinks as well once it sees there's no corresponding tombstones. So for each pool path defined in your /etc/greyhole.conf: `rm -r /path/to/pool/drive/gh/share_name/path_to_remove/*` `rm -r /path/to/pool/drive/gh/.gh_graveyard/share_name/path_to_remove/*` `rm -r /path/to/pool/drive/gh/.gh_graveyard_backup/share_name/path_to_remove/*` The -r above will recursively delete [i]everything[/i] under the path you specify. Some explanation of the paths above: /path/to/pool/drive would be something like /var/hda/files/drives/drive1 share_name would be something like Users path_to_remove would be something like John/Preload (E)/Documents and Settings [b]Note:[/b] be careful if you choose to run the above that you only remove the data you want to delete, obviously going up a directory level could result in deleting a lot more data than you intended!

I hear you on the finer points of rm... and -r... and oh by the way... how about \rm -r.. that one'll bite you if you're not judicious in it's usage... Thanks for all your insight!

In reply to a couple of your points:

I think the better question is whether Greyhole would crash if it can't write to its logs. I'm not sure if it will, I haven't run that scenario before. You can verify when/if you hit this again, though. As root you can run `service greyhole status` if it's not running, then obviously it won't be spooling / processing tasks. If it continues running when logs cannot be written to and your landing zone is on the full partition I see further complications arising.

I had run a 'service greyhole status" and it showed as running throughout the time this issue occurred...
Also... landing zone is on another partition... already learned that lesson.

I've already gone in and deleted some of the /var/hda/files/drives/drivex/gh/Users/UserX/OldStuff/Idon't want files... Based on your suggestion, I need to go back and hit the .gh_graveyard stuff as well.... Once I've completed that, I"m not going to do anything till I see greyhole has no more tasks... then I'll go about deleting more stuff... I think part of the problem was, I went into the share and started cleaning stuff up when greyhole was in the middle of trying to do it's thing for the initial transfer of the data... so moving things around and deleting things before it even had a chance to distribute the data across the pool...

By the way, my suggestion to the powers that be, would be to insert a blurb wherever there are faq/instructions/wiki whatever on drive partitioning... to consider system partition size with regard to situations like these... I'm not sure what the suggested size should be, but I'll never partition an Amahi system partition this small again.. I think I'd go with 100g just to be safe.. (disk is cheap) and I've already got /home and /var/hda/files/ and /media/TV (SageTV) on separate partitions... I know I'm a power user I'm pushing greyhole with the volume of data, but I just didn't see this one coming...

Regards,

jbmia