Greyhole daemon is currently stopped

[stevenjturnbull]

The greyhole service is refusing to start. I tried the troubleshooting steps from the wiki (clearing work queue) as well as the last couple posts here in the forums where the service wouldn't start. Greyhole was set to auto-update until about a week ago but I removed the repo when I got tired of having to recreate the symbolic link when it did. The landing zone filled up but I freed up space and the log is still spitting out an error I can't seem to google an answer for. I recently added a new drive and noticed the problem when I came back a few hours after launching a balance only to see nothing had moved. In the process of backing up a few of the smaller shares so I can make more space in /var/hda/files/gh in case a little more free space will allow it to work.

Thanks for the help.

The greyhole log is full of the following:

Code: Select all

Jan 14 16:56:07 6 daemon: Optimizing MySQL tables... Jan 14 16:56:07 6 daemon: Moving /var/hda/files/gh/.gh_graveyard_backup to /var/hda/files/gh/.gh_metastore_backup... Jan 14 16:56:07 6 daemon: Moving /var/hda/files/drives/drive4/gh/.gh_graveyard_backup to /var/hda/files/drives/drive4/gh/.gh_metastore_backup... Jan 14 16:56:07 2 daemon: Can't rename setting 'graveyard_backup_directory' to 'metastore_backup_directory': Duplicate entry 'metastore_backup_directory' for key 'PRIMARY' Jan 14 16:58:07 6 daemon: Greyhole (version 0.9.25) daemon started. Jan 14 16:58:07 6 daemon: Optimizing MySQL tables... Jan 14 16:58:07 6 daemon: Moving /var/hda/files/gh/.gh_graveyard_backup to /var/hda/files/gh/.gh_metastore_backup... Jan 14 16:58:07 6 daemon: Moving /var/hda/files/drives/drive4/gh/.gh_graveyard_backup to /var/hda/files/drives/drive4/gh/.gh_metastore_backup... Jan 14 16:58:07 2 daemon: Can't rename setting 'graveyard_backup_directory' to 'metastore_backup_directory': Duplicate entry 'metastore_backup_directory' for key 'PRIMARY'

1. What version of Fedora, Samba & Greyhole are you running?

Code: Select all

[root@tmedia sturn88145]# uname -r; rpm -q samba hda-greyhole 2.6.35.14-106.fc14.x86_64 samba-3.5.11-79.fc14.x86_64 hda-greyhole-0.9.25-1.x86_64

2. The content of the /etc/samba/smb.conf & /etc/greyhole.conf files (provide paste URLs):

fpaste /etc/samba/smb.conf
fpaste /etc/greyhole.conf​

3. The result of the following commands:

mount

Code: Select all

/dev/mapper/vg_tmedia-lv_root on / type ext4 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw) /dev/sdf1 on /boot type ext4 (rw) /dev/mapper/vg_tmedia-lv_home on /home type ext4 (rw) /dev/sdf3 on /var/hda/files type ext4 (rw) /dev/sda1 on /var/hda/files/drives/drive1 type ext4 (rw) /dev/sde1 on /var/hda/files/drives/drive2 type ext4 (rw) /dev/sdc1 on /var/hda/files/drives/drive3 type ext4 (rw) /dev/sdb1 on /var/hda/files/drives/drive4 type ext4 (rw) /dev/sdd1 on /var/hda/files/drives/drive5 type ext4 (rw) /dev/sdh1 on /var/hda/files/drives/drive6 type ext4 (rw) /dev/sdi1 on /var/hda/files/drives/drive7 type ext4 (rw) /dev/sdj1 on /var/hda/files/drives/drive8 type ext4 (rw) /dev/sdk1 on /var/hda/files/drives/drive9 type ext4 (rw) /dev/sdg1 on /var/hda/files/drives/drive10 type ext4 (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) fusectl on /sys/fs/fuse/connections type fusectl (rw) gvfs-fuse-daemon on /home/sturn88145/.gvfs type fuse.gvfs-fuse-daemon (rw,nosuid,nodev,user=sturn88145)

fdisk -l

Code: Select all

Disk /dev/sda: 1000.2 GB, 1000204886016 bytes 255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x99455aed Device Boot Start End Blocks Id System /dev/sda1 2048 1953523711 976760832 83 Linux Disk /dev/sdb: 2000.4 GB, 2000398934016 bytes 255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00037c01 Device Boot Start End Blocks Id System /dev/sdb1 2048 3907028991 1953513472 83 Linux Disk /dev/sdc: 1000.2 GB, 1000204886016 bytes 255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0xf177b4fd Device Boot Start End Blocks Id System /dev/sdc1 2048 1953523711 976760832 83 Linux Disk /dev/sdd: 2000.4 GB, 2000398934016 bytes 255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Device Boot Start End Blocks Id System /dev/sdd1 1 3907029167 1953514583+ 83 Linux Disk /dev/sde: 1000.2 GB, 1000204886016 bytes 255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0xf177b4fc Device Boot Start End Blocks Id System /dev/sde1 2048 1953523711 976760832 83 Linux Disk /dev/sdf: 500.1 GB, 500107862016 bytes 255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0xd38aa330 Device Boot Start End Blocks Id System /dev/sdf1 * 2048 1026047 512000 83 Linux /dev/sdf2 1026048 70658047 34816000 8e Linux LVM /dev/sdf3 70658048 976773119 453057536 83 Linux Disk /dev/sdg: 2000.4 GB, 2000398934016 bytes 255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disk identifier: 0x000bcbd5 Device Boot Start End Blocks Id System /dev/sdg1 2048 3907028991 1953513472 83 Linux Disk /dev/sdh: 2000.4 GB, 2000398934016 bytes 255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x0009780d Device Boot Start End Blocks Id System /dev/sdh1 2048 3907028991 1953513472 83 Linux Disk /dev/sdi: 2000.4 GB, 2000398934016 bytes 255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disk identifier: 0x000113e0 Device Boot Start End Blocks Id System /dev/sdi1 2048 3907028991 1953513472 83 Linux Disk /dev/sdj: 2000.4 GB, 2000398934016 bytes 255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x0009089f Device Boot Start End Blocks Id System /dev/sdj1 2048 3907028991 1953513472 83 Linux WARNING: GPT (GUID Partition Table) detected on '/dev/sdk'! The util fdisk doesn't support GPT. Use GNU Parted. Disk /dev/sdk: 3000.6 GB, 3000592982016 bytes 255 heads, 63 sectors/track, 364801 cylinders, total 5860533168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disk identifier: 0x00000000 Device Boot Start End Blocks Id System /dev/sdk1 1 4294967295 2147483647+ ee GPT Partition 1 does not start on physical sector boundary. Disk /dev/dm-0: 4294 MB, 4294967296 bytes 255 heads, 63 sectors/track, 522 cylinders, total 8388608 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/dm-0 doesn't contain a valid partition table Disk /dev/dm-1: 20.8 GB, 20837302272 bytes 255 heads, 63 sectors/track, 2533 cylinders, total 40697856 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/dm-1 doesn't contain a valid partition table Disk /dev/dm-2: 10.5 GB, 10502537216 bytes 255 heads, 63 sectors/track, 1276 cylinders, total 20512768 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/dm-2 doesn't contain a valid partition table

df -h

Code: Select all

Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_tmedia-lv_root 20G 6.0G 13G 33% / tmpfs 3.8G 188K 3.8G 1% /dev/shm /dev/sdf1 485M 68M 393M 15% /boot /dev/mapper/vg_tmedia-lv_home 9.7G 364M 8.8G 4% /home /dev/sdf3 426G 404G 30M 100% /var/hda/files /dev/sda1 917G 852G 19G 98% /var/hda/files/drives/drive1 /dev/sde1 917G 846G 25G 98% /var/hda/files/drives/drive2 /dev/sdc1 917G 845G 26G 98% /var/hda/files/drives/drive3 /dev/sdb1 1.8T 1.7T 22G 99% /var/hda/files/drives/drive4 /dev/sdd1 1.8T 1.7T 25G 99% /var/hda/files/drives/drive5 /dev/sdh1 1.8T 1.7T 25G 99% /var/hda/files/drives/drive6 /dev/sdi1 1.8T 1.7T 24G 99% /var/hda/files/drives/drive7 /dev/sdj1 1.8T 1.7T 25G 99% /var/hda/files/drives/drive8 /dev/sdk1 2.8T 2.6T 26G 100% /var/hda/files/drives/drive9 /dev/sdg1 1.8T 196M 1.7T 1% /var/hda/files/drives/drive10

greyhole --stats​

Code: Select all

Greyhole Statistics =================== Storage Pool Total - Used = Free + Trash = Possible /var/hda/files/gh: 425G - 404G = 0G + 0G = 0G /var/hda/files/drives/drive1/gh: 917G - 851G = 19G + 0G = 19G /var/hda/files/drives/drive2/gh: 917G - 846G = 24G + 0G = 24G /var/hda/files/drives/drive3/gh: 917G - 844G = 26G + 0G = 26G /var/hda/files/drives/drive4/gh: 1834G - 1719G = 21G + 0G = 21G /var/hda/files/drives/drive5/gh: 1834G - 1716G = 25G + 0G = 25G /var/hda/files/drives/drive6/gh: 1834G - 1717G = 24G + 0G = 24G /var/hda/files/drives/drive7/gh: 1834G - 1717G = 24G + 0G = 24G /var/hda/files/drives/drive8/gh: 1834G - 1716G = 24G + 0G = 24G /var/hda/files/drives/drive9/gh: 2792G - 2628G = 25G + 0G = 25G /var/hda/files/drives/drive10/gh: 1834G - 0G = 1740G + 0G = 1740G ========================================== Total: 16971G - 14158G = 1953G + 0G = 1953G

4. The list drives in your storage pool (per Amahi platform):

Code: Select all

+----+-------------------------------+--------------+---------------------+---------------------+ | id | path | minimum_free | created_at | updated_at | +----+-------------------------------+--------------+---------------------+---------------------+ | 1 | /var/hda/files | 10 | 2011-04-18 21:44:02 | 2011-04-18 21:44:02 | | 2 | /var/hda/files/drives/drive1 | 10 | 2011-04-18 22:48:40 | 2011-04-18 22:48:40 | | 3 | /var/hda/files/drives/drive2 | 10 | 2011-04-18 23:25:39 | 2011-04-18 23:25:39 | | 4 | /var/hda/files/drives/drive3 | 10 | 2011-04-19 10:44:03 | 2011-04-19 10:44:03 | | 5 | /var/hda/files/drives/drive4 | 10 | 2011-05-11 00:55:37 | 2011-05-11 00:55:37 | | 6 | /var/hda/files/drives/drive5 | 10 | 2011-06-21 21:19:51 | 2011-06-21 21:19:51 | | 7 | /var/hda/files/drives/drive6 | 10 | 2012-04-17 23:40:26 | 2012-04-17 23:40:26 | | 8 | /var/hda/files/drives/drive7 | 10 | 2012-04-17 23:40:27 | 2012-04-17 23:40:27 | | 9 | /var/hda/files/drives/drive8 | 10 | 2012-06-20 22:40:35 | 2012-06-20 22:40:35 | | 10 | /var/hda/files/drives/drive9 | 10 | 2012-09-25 21:35:51 | 2012-09-25 21:35:51 | | 11 | /var/hda/files/drives/drive10 | 10 | 2013-01-14 17:11:38 | 2013-01-14 17:11:38 | +----+-------------------------------+--------------+---------------------+---------------------+

5. A list of the directories on the root of the drives included in your storage pool, obtained with the following command (provide a paste URL):

http://fpaste.org/ZLvs/

6. The Greyhole work queue:

Code: Select all

Greyhole Work Queue Statistics ============================== This table gives you the number of pending operations queued for the Greyhole daemon, per share. Write Delete Rename Check Backups 0 0 0 0 Books 0 0 0 0 Docs 0 0 0 0 Dropbox 0 0 0 0 Movies 0 0 0 0 Music 0 0 0 0 Music Videos 0 0 0 0 Other Movies 0 0 0 0 Pictures 0 0 0 0 Software 0 0 0 0 Television 0 0 0 0 ================================================ Total 0 0 0 0 Write Delete Rename Check The following is the number of pending operations that the Greyhole daemon still needs to parse. Until it does, the nature of those operations is unknown. Spooled operations that have been parsed will be listed above and disappear from the count below. Spooled 21691

tail /var/log/greyhole.log​

Code: Select all

Jan 14 17:08:07 6 daemon: Greyhole (version 0.9.25) daemon started. Jan 14 17:08:07 6 daemon: Optimizing MySQL tables... Jan 14 17:08:07 6 daemon: Moving /var/hda/files/gh/.gh_graveyard_backup to /var/hda/files/gh/.gh_metastore_backup... Jan 14 17:08:07 6 daemon: Moving /var/hda/files/drives/drive4/gh/.gh_graveyard_backup to /var/hda/files/drives/drive4/gh/.gh_metastore_backup... Jan 14 17:08:07 2 daemon: Can't rename setting 'graveyard_backup_directory' to 'metastore_backup_directory': Duplicate entry 'metastore_backup_directory' for key 'PRIMARY' Jan 14 17:10:07 6 daemon: Greyhole (version 0.9.25) daemon started. Jan 14 17:10:07 6 daemon: Optimizing MySQL tables... Jan 14 17:10:07 6 daemon: Moving /var/hda/files/gh/.gh_graveyard_backup to /var/hda/files/gh/.gh_metastore_backup... Jan 14 17:10:07 6 daemon: Moving /var/hda/files/drives/drive4/gh/.gh_graveyard_backup to /var/hda/files/drives/drive4/gh/.gh_metastore_backup... Jan 14 17:10:07 2 daemon: Can't rename setting 'graveyard_backup_directory' to 'metastore_backup_directory': Duplicate entry 'metastore_backup_directory' for key 'PRIMARY'

[bigfoot65]

I would say to check the permissions of the gh folders. Also there is a newer version available. You might wanna upgrade to the latest. I am running it on Fedora and no issues.

[stevenjturnbull]

I loaded up myphpadmin and renamed metastore_backup_directory to a backup value. Service started right up and everything is working now. No clue what caused the problem in the first place.