Greyhole daemon is currently stopped
Posted: Mon Jan 14, 2013 3:18 pm
The greyhole service is refusing to start. I tried the troubleshooting steps from the wiki (clearing work queue) as well as the last couple posts here in the forums where the service wouldn't start. Greyhole was set to auto-update until about a week ago but I removed the repo when I got tired of having to recreate the symbolic link when it did. The landing zone filled up but I freed up space and the log is still spitting out an error I can't seem to google an answer for. I recently added a new drive and noticed the problem when I came back a few hours after launching a balance only to see nothing had moved. In the process of backing up a few of the smaller shares so I can make more space in /var/hda/files/gh in case a little more free space will allow it to work.
Thanks for the help.
The greyhole log is full of the following:
1. What version of Fedora, Samba & Greyhole are you running?
2. The content of the /etc/samba/smb.conf & /etc/greyhole.conf files (provide paste URLs):
fpaste /etc/samba/smb.conf
fpaste /etc/greyhole.conf
3. The result of the following commands:
mount
fdisk -l
df -h
greyhole --stats
4. The list drives in your storage pool (per Amahi platform):
5. A list of the directories on the root of the drives included in your storage pool, obtained with the following command (provide a paste URL):
http://fpaste.org/ZLvs/
6. The Greyhole work queue:
tail /var/log/greyhole.log
Thanks for the help.
The greyhole log is full of the following:
Code: Select all
Jan 14 16:56:07 6 daemon: Optimizing MySQL tables...
Jan 14 16:56:07 6 daemon: Moving /var/hda/files/gh/.gh_graveyard_backup to /var/hda/files/gh/.gh_metastore_backup...
Jan 14 16:56:07 6 daemon: Moving /var/hda/files/drives/drive4/gh/.gh_graveyard_backup to /var/hda/files/drives/drive4/gh/.gh_metastore_backup...
Jan 14 16:56:07 2 daemon: Can't rename setting 'graveyard_backup_directory' to 'metastore_backup_directory': Duplicate entry 'metastore_backup_directory' for key 'PRIMARY'
Jan 14 16:58:07 6 daemon: Greyhole (version 0.9.25) daemon started.
Jan 14 16:58:07 6 daemon: Optimizing MySQL tables...
Jan 14 16:58:07 6 daemon: Moving /var/hda/files/gh/.gh_graveyard_backup to /var/hda/files/gh/.gh_metastore_backup...
Jan 14 16:58:07 6 daemon: Moving /var/hda/files/drives/drive4/gh/.gh_graveyard_backup to /var/hda/files/drives/drive4/gh/.gh_metastore_backup...
Jan 14 16:58:07 2 daemon: Can't rename setting 'graveyard_backup_directory' to 'metastore_backup_directory': Duplicate entry 'metastore_backup_directory' for key 'PRIMARY'
Code: Select all
[root@tmedia sturn88145]# uname -r; rpm -q samba hda-greyhole
2.6.35.14-106.fc14.x86_64
samba-3.5.11-79.fc14.x86_64
hda-greyhole-0.9.25-1.x86_64
fpaste /etc/samba/smb.conf
fpaste /etc/greyhole.conf
3. The result of the following commands:
mount
Code: Select all
/dev/mapper/vg_tmedia-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sdf1 on /boot type ext4 (rw)
/dev/mapper/vg_tmedia-lv_home on /home type ext4 (rw)
/dev/sdf3 on /var/hda/files type ext4 (rw)
/dev/sda1 on /var/hda/files/drives/drive1 type ext4 (rw)
/dev/sde1 on /var/hda/files/drives/drive2 type ext4 (rw)
/dev/sdc1 on /var/hda/files/drives/drive3 type ext4 (rw)
/dev/sdb1 on /var/hda/files/drives/drive4 type ext4 (rw)
/dev/sdd1 on /var/hda/files/drives/drive5 type ext4 (rw)
/dev/sdh1 on /var/hda/files/drives/drive6 type ext4 (rw)
/dev/sdi1 on /var/hda/files/drives/drive7 type ext4 (rw)
/dev/sdj1 on /var/hda/files/drives/drive8 type ext4 (rw)
/dev/sdk1 on /var/hda/files/drives/drive9 type ext4 (rw)
/dev/sdg1 on /var/hda/files/drives/drive10 type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
fusectl on /sys/fs/fuse/connections type fusectl (rw)
gvfs-fuse-daemon on /home/sturn88145/.gvfs type fuse.gvfs-fuse-daemon (rw,nosuid,nodev,user=sturn88145)
Code: Select all
Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x99455aed
Device Boot Start End Blocks Id System
/dev/sda1 2048 1953523711 976760832 83 Linux
Disk /dev/sdb: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00037c01
Device Boot Start End Blocks Id System
/dev/sdb1 2048 3907028991 1953513472 83 Linux
Disk /dev/sdc: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xf177b4fd
Device Boot Start End Blocks Id System
/dev/sdc1 2048 1953523711 976760832 83 Linux
Disk /dev/sdd: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/sdd1 1 3907029167 1953514583+ 83 Linux
Disk /dev/sde: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xf177b4fc
Device Boot Start End Blocks Id System
/dev/sde1 2048 1953523711 976760832 83 Linux
Disk /dev/sdf: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xd38aa330
Device Boot Start End Blocks Id System
/dev/sdf1 * 2048 1026047 512000 83 Linux
/dev/sdf2 1026048 70658047 34816000 8e Linux LVM
/dev/sdf3 70658048 976773119 453057536 83 Linux
Disk /dev/sdg: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x000bcbd5
Device Boot Start End Blocks Id System
/dev/sdg1 2048 3907028991 1953513472 83 Linux
Disk /dev/sdh: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0009780d
Device Boot Start End Blocks Id System
/dev/sdh1 2048 3907028991 1953513472 83 Linux
Disk /dev/sdi: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x000113e0
Device Boot Start End Blocks Id System
/dev/sdi1 2048 3907028991 1953513472 83 Linux
Disk /dev/sdj: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0009089f
Device Boot Start End Blocks Id System
/dev/sdj1 2048 3907028991 1953513472 83 Linux
WARNING: GPT (GUID Partition Table) detected on '/dev/sdk'! The util fdisk doesn't support GPT. Use GNU Parted.
Disk /dev/sdk: 3000.6 GB, 3000592982016 bytes
255 heads, 63 sectors/track, 364801 cylinders, total 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/sdk1 1 4294967295 2147483647+ ee GPT
Partition 1 does not start on physical sector boundary.
Disk /dev/dm-0: 4294 MB, 4294967296 bytes
255 heads, 63 sectors/track, 522 cylinders, total 8388608 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/dm-0 doesn't contain a valid partition table
Disk /dev/dm-1: 20.8 GB, 20837302272 bytes
255 heads, 63 sectors/track, 2533 cylinders, total 40697856 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/dm-1 doesn't contain a valid partition table
Disk /dev/dm-2: 10.5 GB, 10502537216 bytes
255 heads, 63 sectors/track, 1276 cylinders, total 20512768 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/dm-2 doesn't contain a valid partition table
Code: Select all
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_tmedia-lv_root
20G 6.0G 13G 33% /
tmpfs 3.8G 188K 3.8G 1% /dev/shm
/dev/sdf1 485M 68M 393M 15% /boot
/dev/mapper/vg_tmedia-lv_home
9.7G 364M 8.8G 4% /home
/dev/sdf3 426G 404G 30M 100% /var/hda/files
/dev/sda1 917G 852G 19G 98% /var/hda/files/drives/drive1
/dev/sde1 917G 846G 25G 98% /var/hda/files/drives/drive2
/dev/sdc1 917G 845G 26G 98% /var/hda/files/drives/drive3
/dev/sdb1 1.8T 1.7T 22G 99% /var/hda/files/drives/drive4
/dev/sdd1 1.8T 1.7T 25G 99% /var/hda/files/drives/drive5
/dev/sdh1 1.8T 1.7T 25G 99% /var/hda/files/drives/drive6
/dev/sdi1 1.8T 1.7T 24G 99% /var/hda/files/drives/drive7
/dev/sdj1 1.8T 1.7T 25G 99% /var/hda/files/drives/drive8
/dev/sdk1 2.8T 2.6T 26G 100% /var/hda/files/drives/drive9
/dev/sdg1 1.8T 196M 1.7T 1% /var/hda/files/drives/drive10
Code: Select all
Greyhole Statistics
===================
Storage Pool
Total - Used = Free + Trash = Possible
/var/hda/files/gh: 425G - 404G = 0G + 0G = 0G
/var/hda/files/drives/drive1/gh: 917G - 851G = 19G + 0G = 19G
/var/hda/files/drives/drive2/gh: 917G - 846G = 24G + 0G = 24G
/var/hda/files/drives/drive3/gh: 917G - 844G = 26G + 0G = 26G
/var/hda/files/drives/drive4/gh: 1834G - 1719G = 21G + 0G = 21G
/var/hda/files/drives/drive5/gh: 1834G - 1716G = 25G + 0G = 25G
/var/hda/files/drives/drive6/gh: 1834G - 1717G = 24G + 0G = 24G
/var/hda/files/drives/drive7/gh: 1834G - 1717G = 24G + 0G = 24G
/var/hda/files/drives/drive8/gh: 1834G - 1716G = 24G + 0G = 24G
/var/hda/files/drives/drive9/gh: 2792G - 2628G = 25G + 0G = 25G
/var/hda/files/drives/drive10/gh: 1834G - 0G = 1740G + 0G = 1740G
==========================================
Total: 16971G - 14158G = 1953G + 0G = 1953G
Code: Select all
+----+-------------------------------+--------------+---------------------+---------------------+
| id | path | minimum_free | created_at | updated_at |
+----+-------------------------------+--------------+---------------------+---------------------+
| 1 | /var/hda/files | 10 | 2011-04-18 21:44:02 | 2011-04-18 21:44:02 |
| 2 | /var/hda/files/drives/drive1 | 10 | 2011-04-18 22:48:40 | 2011-04-18 22:48:40 |
| 3 | /var/hda/files/drives/drive2 | 10 | 2011-04-18 23:25:39 | 2011-04-18 23:25:39 |
| 4 | /var/hda/files/drives/drive3 | 10 | 2011-04-19 10:44:03 | 2011-04-19 10:44:03 |
| 5 | /var/hda/files/drives/drive4 | 10 | 2011-05-11 00:55:37 | 2011-05-11 00:55:37 |
| 6 | /var/hda/files/drives/drive5 | 10 | 2011-06-21 21:19:51 | 2011-06-21 21:19:51 |
| 7 | /var/hda/files/drives/drive6 | 10 | 2012-04-17 23:40:26 | 2012-04-17 23:40:26 |
| 8 | /var/hda/files/drives/drive7 | 10 | 2012-04-17 23:40:27 | 2012-04-17 23:40:27 |
| 9 | /var/hda/files/drives/drive8 | 10 | 2012-06-20 22:40:35 | 2012-06-20 22:40:35 |
| 10 | /var/hda/files/drives/drive9 | 10 | 2012-09-25 21:35:51 | 2012-09-25 21:35:51 |
| 11 | /var/hda/files/drives/drive10 | 10 | 2013-01-14 17:11:38 | 2013-01-14 17:11:38 |
+----+-------------------------------+--------------+---------------------+---------------------+
http://fpaste.org/ZLvs/
6. The Greyhole work queue:
Code: Select all
Greyhole Work Queue Statistics
==============================
This table gives you the number of pending operations queued for the Greyhole daemon, per share.
Write Delete Rename Check
Backups 0 0 0 0
Books 0 0 0 0
Docs 0 0 0 0
Dropbox 0 0 0 0
Movies 0 0 0 0
Music 0 0 0 0
Music Videos 0 0 0 0
Other Movies 0 0 0 0
Pictures 0 0 0 0
Software 0 0 0 0
Television 0 0 0 0
================================================
Total 0 0 0 0
Write Delete Rename Check
The following is the number of pending operations that the Greyhole daemon still needs to parse.
Until it does, the nature of those operations is unknown.
Spooled operations that have been parsed will be listed above and disappear from the count below.
Spooled 21691
Code: Select all
Jan 14 17:08:07 6 daemon: Greyhole (version 0.9.25) daemon started.
Jan 14 17:08:07 6 daemon: Optimizing MySQL tables...
Jan 14 17:08:07 6 daemon: Moving /var/hda/files/gh/.gh_graveyard_backup to /var/hda/files/gh/.gh_metastore_backup...
Jan 14 17:08:07 6 daemon: Moving /var/hda/files/drives/drive4/gh/.gh_graveyard_backup to /var/hda/files/drives/drive4/gh/.gh_metastore_backup...
Jan 14 17:08:07 2 daemon: Can't rename setting 'graveyard_backup_directory' to 'metastore_backup_directory': Duplicate entry 'metastore_backup_directory' for key 'PRIMARY'
Jan 14 17:10:07 6 daemon: Greyhole (version 0.9.25) daemon started.
Jan 14 17:10:07 6 daemon: Optimizing MySQL tables...
Jan 14 17:10:07 6 daemon: Moving /var/hda/files/gh/.gh_graveyard_backup to /var/hda/files/gh/.gh_metastore_backup...
Jan 14 17:10:07 6 daemon: Moving /var/hda/files/drives/drive4/gh/.gh_graveyard_backup to /var/hda/files/drives/drive4/gh/.gh_metastore_backup...
Jan 14 17:10:07 2 daemon: Can't rename setting 'graveyard_backup_directory' to 'metastore_backup_directory': Duplicate entry 'metastore_backup_directory' for key 'PRIMARY'