RESOLVED: Greyhole not showing queue

[aarondyck]

So I'm quite happy with the Greyhole platform, but lately I've run into an issue where it isn't showing me what is in the queue.

When I issue the command greyhole -q the system just hangs until I issue CTRL+C.

All other greyhole commands seem to be working. My shares are mounted:

Code: Select all

[root@localhost aaron]# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda6 897G 713G 138G 84% / devtmpfs 1.7G 0 1.7G 0% /dev tmpfs 1.7G 0 1.7G 0% /dev/shm tmpfs 1.7G 816K 1.7G 1% /run tmpfs 1.7G 0 1.7G 0% /sys/fs/cgroup tmpfs 1.7G 47M 1.7G 3% /tmp /dev/sdc1 917G 491G 380G 57% /var/hda/files/drives/drive4 /dev/sdb1 917G 501G 370G 58% /var/hda/files/drives/drive3 /dev/sda1 190M 60M 117M 34% /boot /dev/sde1 2.7T 2.2T 372G 86% /var/hda/files/drives/drive2 /dev/sda2 14G 193M 13G 2% /home /dev/sda5 3.3G 6.8M 3.1G 1% /var/hda/files/drives/drive1 none 4.0M 0 4.0M 0% /var/spool/greyhole/mem //127.0.0.1/tv 4.5T 3.4T 1.1T 76% /mnt/samba/TV //127.0.0.1/anime 4.5T 3.4T 1.1T 76% /mnt/samba/Anime //127.0.0.1/books 4.5T 3.4T 1.1T 76% /mnt/samba/Books //127.0.0.1/movies 4.5T 3.4T 1.1T 76% /mnt/samba/Movies //127.0.0.1/music 4.5T 3.4T 1.1T 76% /mnt/samba/Music //127.0.0.1/pictures 4.5T 3.4T 1.1T 76% /mnt/samba/Pictures //127.0.0.1/software 4.5T 3.4T 1.1T 76% /mnt/samba/Software

I'm not sure if it's even trying to balance anything since it's not showing me the queue.

Status seems okay:

Code: Select all

[root@localhost aaron]# greyhole -s Greyhole Statistics =================== Storage Pool Total - Used = Free + Trash = Possible /var/hda/files/drives/drive2/gh: 2751G - 2239G = 372G + 0G = 372G /var/hda/files/drives/drive3/gh: 917G - 501G = 369G + 0G = 369G /var/hda/files/drives/drive4/gh: 917G - 490G = 380G + 0G = 380G ========================================== Total: 4584G - 3230G = 1121G + 0G = 1121G

Systemctl status looks ... okay?

Code: Select all

[root@localhost aaron]# systemctl status amahi-greyhole.service amahi-greyhole.service - Greyhole for Amahi Loaded: loaded (/usr/lib/systemd/system/amahi-greyhole.service; enabled) Active: active (running) since Fri 2015-03-13 22:03:19 EDT; 3min 40s ago Process: 1973 ExecStopPost=/bin/rm -f /run/amahi-greyhole.pid (code=exited, status=0/SUCCESS) Process: 1971 ExecStop=/usr/bin/kill $MAINPID (code=exited, status=0/SUCCESS) Process: 1975 ExecStart=/usr/bin/amahi-greyhole (code=exited, status=0/SUCCESS) Main PID: 1976 (greyhole) CGroup: name=systemd:/system/amahi-greyhole.service ├─1976 /usr/bin/php -d open_basedir=/ /usr/bin/greyhole --daemon ├─2045 sh -c find -L /var/spool/greyhole -type f -printf "%T@ %p\n" | sort -n 2> /dev/null | head -10000000 ├─2046 find -L /var/spool/greyhole -type f -printf %T@ %p n ├─2047 sort -n └─2048 head -10000000 Mar 13 22:03:19 localhost.localdomain systemd[1]: Starting Greyhole for Amahi... Mar 13 22:03:19 localhost.localdomain systemd[1]: Started Greyhole for Amahi.

And the tail of the greyhole log definitely doesn't show what I expect (I expect to see Nothing to do...sleeping):

Code: Select all

[root@localhost aaron]# tail /var/log/greyhole.log Mar 13 21:57:44 DEBUG daemon: Loading metadata store backup directories... Mar 13 21:57:44 DEBUG daemon: Found 2 directories in the settings table. Mar 13 21:57:44 DEBUG daemon: Checking symlink at /usr/lib64/samba/vfs/greyhole.so... Mar 13 21:57:44 DEBUG daemon: Is OK. Mar 13 22:03:19 INFO daemon: Greyhole (version 0.9.60) daemon started. Mar 13 22:03:19 INFO daemon: Optimizing MySQL tables... Mar 13 22:03:19 DEBUG daemon: Loading metadata store backup directories... Mar 13 22:03:19 DEBUG daemon: Found 2 directories in the settings table. Mar 13 22:03:19 DEBUG daemon: Checking symlink at /usr/lib64/samba/vfs/greyhole.so... Mar 13 22:03:19 DEBUG daemon: Is OK.

Any thoughts?

[bigfoot65]

Not sure what the issue might be here.

How long do you wait on the

Code: Select all

greyhole -q

command before you interrupt it? Maybe it just needs longer.

Did you restart Greyhole? The log reflects a status like it was just restarted in my opinion.

I am no expert on Greyhole, so maybe the author will jump in and provide assistance.

[aarondyck]

I've let it run for a few minutes. I have done a lot of troubleshooting already, including clearing the task table (0 lines affected).

I just started an fsck so I'm going to let that run for the night to see if it helps. I'll also run greyhole -q and let it run overnight as well to see if I do eventually get results.

I'll post an update in the morning.

[aarondyck]

So it's been over an hour since I started the fsck and ran greyhole -q

Here's the tail of the log:

Code: Select all

[aaron@localhost ~]$ tail -f /var/log/greyhole.log Mar 13 22:03:19 DEBUG daemon: Checking symlink at /usr/lib64/samba/vfs/greyhole.so... Mar 13 22:03:19 DEBUG daemon: Is OK. Mar 13 23:09:22 ERROR read_smb_spool: PHP Fatal Error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 32 bytes); BT: greyhole[L4614] Mar 13 23:09:25 INFO daemon: Greyhole (version 0.9.60) daemon started. Mar 13 23:09:25 INFO daemon: Optimizing MySQL tables... Mar 13 23:09:25 DEBUG daemon: Loading metadata store backup directories... Mar 13 23:09:25 DEBUG daemon: Found 2 directories in the settings table. Mar 13 23:09:25 DEBUG daemon: Checking symlink at /usr/lib64/samba/vfs/greyhole.so... Mar 13 23:09:25 DEBUG daemon: Is OK. Mar 13 23:21:33 INFO fsck: Cleaning executed tasks: keeping the last 60 days of logs.

And still no response from greyhole -q.

I'm letting both continue to run, but it doesn't look promising.

[aarondyck]

Tasks shows the fsck:

Code: Select all

MariaDB [greyhole]> select * from tasks; +----+--------+-------------------------+-----------+-----------------+----------+---------------------+ | id | action | share | full_path | additional_info | complete | event_date | +----+--------+-------------------------+-----------+-----------------+----------+---------------------+ | 1 | fsck | /var/hda/files/music | NULL | NULL | yes | 2015-03-13 23:21:33 | | 2 | fsck | /var/hda/files/pictures | NULL | NULL | yes | 2015-03-13 23:21:33 | | 3 | fsck | /var/hda/files/software | NULL | NULL | yes | 2015-03-13 23:21:33 | | 4 | fsck | /var/hda/files/movies | NULL | NULL | yes | 2015-03-13 23:21:33 | | 5 | fsck | /var/hda/files/tv | NULL | NULL | yes | 2015-03-13 23:21:33 | | 6 | fsck | /var/hda/files/anime | NULL | NULL | yes | 2015-03-13 23:21:33 | | 7 | fsck | /var/hda/files/books | NULL | NULL | yes | 2015-03-13 23:21:33 | +----+--------+-------------------------+-----------+-----------------+----------+---------------------+ 7 rows in set (0.17 sec)

Nothing in tasks_completed:

Code: Select all

MariaDB [greyhole]> select * from tasks_completed; Empty set (0.04 sec) MariaDB [greyhole]>

[aarondyck]

It looks like the greyhole -q did complete last night at some point. I'm a bit concerned by the results, though:

Code: Select all

[aaron@localhost ~]$ greyhole -q Greyhole Work Queue Statistics ============================== This table gives you the number of pending operations queued for the Greyhole daemon, per share. Write Delete Rename Check Anime 0 0 0 0 Books 0 0 0 0 Movies 0 0 0 0 Music 0 0 0 0 Pictures 0 0 0 0 Software 0 0 0 0 TV 0 0 0 0 ============================================ Total 0 0 0 0 Write Delete Rename Check The following is the number of pending operations that the Greyhole daemon still needs to parse. Until it does, the nature of those operations is unknown. Spooled operations that have been parsed will be listed above and disappear from the count below. Spooled 6,846,871

[bigfoot65]

Yes that is odd.

What has changed with your HDA around the time this started being a problem? Are there any apps you installed/reinstalled? OS updates?

[aarondyck]

The only thing I've installed lately was amahi-netboot. I can't confirm if it was working properly before, as I don't go checking my Greyhole queue all the time.

[bigfoot65]

Ok, just thought I would ask.

Have you installed or tried to install the Amahi app VNC ? If so, that may have caused an issue.

Problems with Greyhole occurred with some users, so we have pulled that app from the store. There is a potential solution that corrected their problem we can try if so.

They ended up reinstalling the amahi-greyhole package:

Code: Select all

yum reinstall amahi-greyhole

That fixed the issue for those users.

[aarondyck]

I've never tried using the VNC app.

A reinstall of greyhole doesn't seem to have any effect on the issue.