Greyhole log file.

[luisantonioa]

Hi my server stopped working earlier today due to the root drive running out of space. It's a 74Gb raptor drive and its normally at 7% usage.

However after ssh'ing into the server and looking round i found the that the greyhole log file was 63Gb and had taken up the entire drive..

While I can sort out the problem it had leading up to logging so much data.. I have an issue with the fact that the log file is allowed to grow so big in first place..

Is there a way to restrict the size of the log file?

Cheers

[luisantonioa]

btw I was using amahi 0.9.1-1 but have upgraded it to 0.9.2-1 now

[gboudreau]

No, there is no way to restrict the log file size (not in Greyhole or Amahi anyway).

There are probably Linux tools that can do this though. Asking this (or looking for this) on http://serverfault.com/ should give you the answer you're looking for.

[radioz]

One question: When you looked in the /var/log directory, were there multiple greyhole* log files taking up space, or just a single giant one?

Perhaps give us the output of 'ls -l /var/log/greyhole*'.

If there are multiple greyhole log files, then we can at least restrict the max size of the log files and also use file compression to make them smaller. Take a look at 'man logrotate'. There is a config file in /etc/logrotate.d/ that provides parameters to logrotate that control how logging is done.

Obviously there is a problem if greyhole is generating 65GB log files, but perhaps we can prevent crashing systems by restricting the amount of logging it does.

[gboudreau]

radioz: greyhole comes with it's own logrotate config:
http://greyhole.googlecode.com/svn/trun ... e.greyhole

[radioz]

Yes, I guess I should have stated that explicitly.

Would it make sense to configure logrotate for greyhole to limit the amount of log space it might consume in cases like this?

[luisantonioa]

Hi there were multiple logs files.. i think about 6 or 7 totaling 330Mb but the main one was still massive.. i deleted them all and rebooted the server to get it back up.. i thought i might have to delve into logrotate was just wondering though if there was some explanation for why it spiraled out of control and why there wouldn't be some form of maximum to prevent server failure from running out of room.

Ill have a look into logrotate and trying to get the greyhole config set to a sane maximum so this doesn't repeat.

Cheers for the help.

[gboudreau]

logrotate doesn't handle the current (day) log file, so it can't prevent a single day log from becoming too large. It handle pasts log files. It's configured to keep the latest 7 log files, and compress them, for Greyhole logs.

To know why the log file had become this large, we'd have to know what it contained. A tail -100 /var/log/greyhole.log would have sufficed.

I don't know of any daemon that has an option for a maximum size for log files.
I myself once had samba logs fill my root partition completely.

[evylrat]

I can beat that. Today the log file filled my root partition, 13G the log file was when i deleted it.
I moved my Music collection into another folder, I guess it's moving all the symlinks

Code: Select all

Jul 07 22:23:06 7 rename: Found a broken symlink to update: /var/hda/files/music/_Unsorted/Bass Mekanik - Quad Maximus/Bass Mekanik - Quad Maximus - Bass Mekanik.mp3. Old (broken) target: /var/hda/files/drives/drive2/gh/Music/Albums/Bass Mekanik - Quad Maximus/Bass Mekanik - Quad Maximus - Bass Mekanik.mp3; new (fixed) target: /var/hda/files/drives/drive2/gh/Music/Albums/Bass Mekanik - Quad Maximus/Bass Mekanik - Quad Maximus - Bass Mekanik.mp3 Jul 07 22:23:06 7 rename: Found a broken symlink to update: /var/hda/files/music/_Unsorted/Bass Mekanik - Quad Maximus/Bass Mekanik - Quad Maximus - Bass Zone.mp3. Old (broken) target: /var/hda/files/drives/drive2/gh/Music/Albums/Bass Mekanik - Quad Maximus/Bass Mekanik - Quad Maximus - Bass Zone.mp3; new (fixed) target: /var/hda/files/drives/drive2/gh/Music/Albums/Bass Mekanik - Quad Maximus/Bass Mekanik - Quad Maximus - Bass Zone.mp3 Jul 07 22:23:06 7 rename: Found a broken symlink to update: /var/hda/files/music/_Unsorted/Bass Mekanik - Quad Maximus/Bass Mekanik - Quad Maximus - 40-44 Hz.mp3. Old (broken) target: /var/hda/files/drives/drive2/gh/Music/Albums/Bass Mekanik - Quad Maximus/Bass Mekanik - Quad Maximus - 40-44 Hz.mp3; new (fixed) target: /var/hda/files/drives/drive2/gh/Music/Albums/Bass Mekanik - Quad Maximus/Bass Mekanik - Quad Maximus - 40-44 Hz.mp3 Jul 07 22:23:06 7 rename: Found a broken symlink to update: /var/hda/files/music/_Unsorted/Bass Mekanik - Quad Maximus/Bass Mekanik - Quad Maximus - 85-89 Hz.mp3. Old (broken) target: /var/hda/files/drives/drive2/gh/Music/Albums/Bass Mekanik - Quad Maximus/Bass Mekanik - Quad Maximus - 85-89 Hz.mp3; new (fixed) target: /var/hda/files/drives/drive2/gh/Music/Albums/Bass Mekanik - Quad Maximus/Bass Mekanik - Quad Maximus - 85-89 Hz.mp3

The drive filled up again, I deleted it, and the log file is still spiralling out of control. Within 2 hours it's up to 7Gb already.
Is there no way to disable the log? If I leave this running, it'll just fill the partition again.

[bigfoot65]

See the example greyhole.conf file.

https://gist.github.com/1379102

There is a log line there and you could probably comment it out or point it to a different location with more space.

Seems there must be something strange going on as I have never heard/seen the log file get so large.